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ABSTRACT 


An analysis of the incomprehensibility of large, complex computer systems is made. 
The thesis is that there Is strong relationship between system incomprehensibility 
and the necessity to trust computer systems. A cogent definition of incomprehensi- 
bility In computer systems is established, with common themes drawn from Interdis- 
ciplinary literature dealing with computers and society. Reasons for the creation of 
incomprehensible computer systems are explored, as well as the consequences 
(both technical and social) of using and relying on them. The relationship between 
the real and perceived purposes of computer systems and the appropriateness of. 
trusting these systems is analyzed. Approaches for dealing with the existence of 
vital computer systems which are functionally incomprehensible are evaluated, and 
positive suggestions are made. 
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The technological society contains many parts and specialized 
activities within a myriad of interconnections... The totality of 
such interconnections - the relationships of the parts to each 
other and the. parts to the. whole:- is something.which is no 
longer comprehensible to anyone. In the compiexity of this 
world, people are confronted with extreordinary events and 
functions that are literally unintelligible to them. They are 
unable to give an. adaquate explanation of man. made 
phenomena in their immediate experience. They are unabie to 
form a coherent, rational picture. of: the whole. -Under. these. cir- 
cumstances, all persons do and, indeed, must accept a great 
number of things on faith. ‘They; are-aware thatthe: major conr-. 
ponents of complex systems usually work, that other specialists 
know what they are doing,.and -that: somehow .the. whole fits 
together in relatively good adjustment. Their way of under- 
Standing, however, is basically religious rather shan. sglentific; 
only a small portion of one’s everyday experience in the tech- 
nological society can be made sclentific,.-For the.rest, everyone 
is forced to depend upon and have faith in matters about which 
one has little information or intelligent graep.. 1 -is..thig.condition _ . 
that Ellul describes as the source of the ecoan versions of 
mystery, magic, and the sa R oo 


Langdon Winner 

In recent years, the computer science community has begun to recognize 
comprehensibility as an important dimension of computes programs. Much earlier, 
thoughtful observers of the growing preeminence of technology in modern cultures 
worried about the diminishing ability of people to understand and cope with the 
technological system with which ey. were so ) deeply involved. Today, Dr. Robert 
Johnson, Vice-President for Engineering of Burroughs Gorparation: is not alone in his 
belief that the most serious problem facing the “computer Industry Is the 


2 


incomprehensibility of large computer systems. Several factors combine to make 


computer system incomprehensibility an Issue of immediate « concern to any soclety 


TLangdon Winner, Autonomous Technology (Cambridge, MA: The MIT Press, 
1977), p. 284. 


2Hearsay. 
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as committed to technology as our own: notably, the rapid rate of proliferation of 
new systems, the strong social and economic incentives to use them, and the rising 
level of dependance on computer systems that have vital, nonreversibie impacts on 
our lives. If widespread unintended ‘codmaquencesiot aes computer systems that 
cannot be understoed are to be avoided; the problem of Incomprehensibllity must be 
analyzed and deait with now. This thesis 4s about the incomprehensibility of many 
computer systems we presently use and on which we depend heavily. 

The existing notion of incomprehensibility, as documented in the computer 
science literature as well as in writings from relevant: nontechnical fields (such as 
sociology and poiltical science), is highly ambiguous; tire fees; my. first objective, 
pursued in Chapter 2, Is to sharpen this notion and to arrive at a definition from 
which one may more usefully proceed. Some éimbidt-<akieadbaitn have a tendency 
to become so Involved with the detalis of the easiest ‘matheds of-.rendering com- 
puter programs understandable (for example, the Pectenque? of shuchved program 
ming or the avoidance of goto statements) that they seem to tcrget about more 
general, higher iavai issues. Conversely, critics of Gacnicsoaes social role, 
although they may appreciate the widespread effects of technological innovation, 
are often hampered by the inability to ‘indaretand specific technical applications. 
The study of computer system Inoomprehensibility thus demands an interdisciplinary 
approach, based on an understanding of both eéeiputece ‘iemeaives and the tech 
nological system of which they are ainone the foremost ‘representatives. 
Throughout this thesis, | refer to ‘the ‘observations of diverse groups of 
people - computer scientists, philosophers, ‘peyehoboanete. ‘apeldlogiats. It is not 
expected. that the reader will be famitiar with all the relevant disciplines; therefore, 
| have provided a "list of characters" - a collection of short biographies of most of 


the people on whose ideas | have drawn - following the body of the thesis. 


8. Chepter 1: Introduction 


In discussing computer comprehensibility, it is useful to first consider 
comprehension in a broad context as, for example, in communication between peo 
ple. This need not be a specialized endeavor; most people havea strong intuitive 
sense of what It means to understand someone. elise, and it Is this informally 
learned knowledge that. one projects onto one’s interactions with computers. Shar- 
pening the general concept of comprehension, as. it Is applicable to human communi 
cation, In order to arrive at its much more specific. form, namely as it appears in the 
context of computers, requires competence In computer science. Insights about 
comprehension, when appiled to problem evaluation, _system design, Sfasramaning: 
etc. (all: fundamentally human. activities), can.heip establish what it means to 
comprehend a computer system and the behavior.of computer. systems. 

incomprehenaibility is not a property of .a computer. system (In the sense 
that color is a property of an orange); rather, It is a derived attribute which is 
dependant: upon the context In which.a system.is used and the criteria according 
to which it is judged. An airline reservation system might be crystal clear to the 
reservation clerks who use it avery day, but. largely mysterious to the systems” 
analysts. who attempt to modify it. In addition, incomprehensibility takes on @ 
different meaning in relation to the "front end” of a system (analysis of a problem 
and design of a system) than it does to the “back. end" (utilization and. mainte- 
nance of a system once it has been implemented). Difficulties are bound to arise 
either when a system is designed in a haphazard, ad hoc fashion (surprisingly conr 
mon outside the restricted domain of research systems) or when users are forced 
to communicate with a system withqut some. knowledge af the theory on which its 
-design was based. Indeed, incomprehensible systems. often. turn out to be those 
which are based on-no well formed theory at.all. 


it is important at thie point to stress the fact that.| am concerned with the 
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incomprehensibility that arises from systems; that Is, collections of interrelated and 
intercommunicating activities, of which computers and computer programs are impor- 
tant, but not exclusive, components. By the term “computer system" | wish to 
refer to not just computers themselves, but also the people who choose to design, 
maintain, and use them.. Computer systems bring Into question much more than just 
computers and the programs that run on them; for instance, the nature of the prob- 
fems that we deem sultabie for computerized solutions, and the poorly understood 
processes of problem analysis, system specification and design, and. programming. 

The kind of incomprehensibitity | ‘én interested in-does not derive solely, or 
even mainly, from any easily identifiable: errors (such as coding errors or obscure 
programming), but from more elusive probtems with-the way we think about and deal 
with technology in general, and computers in particular. if we apply computer tech- 
nology Inappropriately or indiscriminately (for tetahce: If we are more motivated by 
ait eagerness to make use of computers than by the: actual effectiveness of apply 
ing computers in a given application), we may end up having difficulty understanding 
the relationship between the original problem and the computer system constructed 
in response to It. In some cases, "problems" are artificially created. or tailored to 
make them better suited for ‘applications of current technology.. Computer systems 
that arise from such situations can be functionally incomprehensible - incomprehen- 
sible in relation to the problem that a system’s users believe it is “solving.” 

Already in the present discussion, | have turned to the question of how does 
incomprehensibility arise in a computer system. in Chapter .3, | examine factors 
which can lead to the generation of incomprehensible computer systems. Concems 
about the process of programming are relevant here: Gerald Weinberg has re- 
minded us that programming is a human activity’ with a psychological component 


which is often ignored, but which significantly ‘affects the quality of programs which 
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are written. The activity of programming a large computer system is plagued by 
what some people have described as problems. of communication. interactions 
between the diverse groups of people who are touched. by computers can be 
stifled by the elitist belief, held by some specialists, that non-technical knowledge 
Is not very relevant in the design of é-computar aystem, or that outsiders shquid 
not question the appropriateness of appliaatians of technology. Professional isola- 
tionism helps distort the. knowledge that .ceders, system. designers, users, and 
everyone in-between can have of a system (by codere, t rafer.to people who carry 
out fairly routine programming jobs which are handed: over to them, by. other people 
who are mote Involved with higher level jobs tike the design of:a system). Bizarre 
stereotypes of programmers as solitary individuala who. are detached from: other 
-people are reinforced by the informal.attitudinal: training of. software workers, which 
tends to discourage curlosity beyond the level of specifin, unconnected, progranr 
thing tasks. i ae aed 

‘In addition to relatively low level problems which may be inherent In the pro- 
gramming process, there are broader, societal issues which. have important. impacts 
on our relationship to computation; these issues ere discussed in Chapter 4. As. ! 
mentioned before, a thorough consideration of: the. .problems entrained by 
incomprehensible computer systems requires -both an understanding of computers 
and a high degree of sensitivity to the social contexts. in which computers play an 
important role. in the modern world, one ought not talk about. social "problems" 
without talking about technology, nor discuss technolegy without taking Into 
account its: social context. 

The present organization of society is such that: there is an air of inevitabil- 
ity about the role of technology. Our ability to critically evaluate social problems 


and proposed computer solutions to these problems Is strongly influenced by what 
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appears to some as our acceptance of the autonomy of technology at the expense 
of our own human autonomy. The perception of technology as an irresistible force 
leads to a situation where the usefulness of the computer is often assumed, even 
when a given application of computer technology is by many criteria Inappropriate; 
where the widespread’ use of computers is accepted.in spite: of. the fears and mis- 
givings of many people; where the surface:.appeal of quick technological fixes for 
pressing probleme often causes a redefinition of our problems to make them more 
amenable to computerized solutions. Present day society uncritically accepts a 
way of life: founded on technical necessity and shaped primarily. by what Langdon 
Winneét calls’ the technical (rational, artificial, productive) made. of activity and 
thought: our love affair with technology has always been-characterized by blind 
ness, and the so-called computer revolution is only the most recent example of. this 
technological preoccupation. 

Chapter 5 of this thesis Is a discussion of the results of the use of 
incomprehensible computer systems end of the social system in which they are 
embedded. Some technical results are the inability to: sates adequate reliability 
for many farge, compiex computer systems, and the resistance of of these systems 
to even minor modifications:: Other consequences are extensions of points | raise. in 
my discussion of potential sources of incomprehensibility; for instance, the obscure- 
tion of the root of a problem as a result of over-rationalizing the processes of 
evaluating problems and of planning solutions :to them. - if a computer. application is 
viewed solely in information processing terms, there may be a gap sensed between 
the problem and the computer system which was designed to..selve it, but which in 


actuality attacks only those symptoms of the. problem which were easy to translate 


Swinner, p. 127. 
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into the form of a computer program. The size of this gap refiects the importance 
of those aspects of the problem which could not be expressed according to. reduc- 
tionist criteria, and hence were never reckoned into the design of the computer 
system. Technological elitism, which.|-clted earlier as a source of incomprehensibit- 
ity, is strengthened by the existence of computer systems which only a few tectr 
nically trained people can plausibly .clalm. to understand or be able to maintain. In 
Kenneth Laudon’s words, we are experiencing " .. . a legitimization of technological 
‘experts’ at the expense of poets."4 

On a still broader level, an attempt to calculate the costs of living in a tech- 
nologically based society must include some estinate of the personal. price that is 
pald - the human suffering. There has been. a. gradual adaptation.of human needs, 
desires, and thought processes to that..which: contemperary technology can explain 
and satisfy. What happens to the ene Teer when. “all the business of 
of view"7® As a result of society's continuing quest for..the perfection of the 
machine, some people now seriously question whether we. willbe able to keep pace 
with our computers. | will-examine thease and. other. unintended, but perhaps ur 
avoidable byproducts. of the computer “revolution.” 

Perhaps. the most.serious effect of the widespread use of computer systems 
is our growing dependance on.computer systems that.we do. not understand. The 
use of computer systems that are: not comprehensible. can result in. the loss of our 


control over the processes that computers monitor, as we become increasingly 


Akenneth Laudon, review of The Conquest of Will: laformation 5 Pretewelng in 
Human Affairs, by Abbe Mowshowitz, Science, 103 (September, 10676), 1111. 


; Sjacques Eltul, The Technological Society (Mew. York: Vintage Books, 1964), 
p. 117. 
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dependant on the information, and even the decisions, which are the output of the 
computers. in actual appii¢ations involving large, complex computer systems, peo- 
ple using such systems’ output often come to rely on "what the machine says." 
They are effectively in a position of either. havirig'to" te the work the computer is 
supposed to be doing, or of having to uncritically accept the output of the con- 
puter. The inherent complexity of many systems makes It almost impossible to ade- 
quatety explain, predict, or trace theif ‘operations. ‘Nevertheless, in using these 
computer systems, one must cross a threshold ‘from: reliance to unjustified trust. 
Trust is deeply embedded with human values; arid it le ‘net at-aif clear that the pro- 
Jection of these values onto maf-machine interactions ts appropriateor. desirable. 

An unwillingness to depend on computers to aid in:important decision making 
may be deemed evidence’ of an “antitechnoldgical™ attitude. Nowadays, an attitude 
of technological optimisin and ‘trust appears fiét to néed justification, while techno 
logical distrust is often graéted with Hostility. Thie sovlal-preseure against eriticism 
‘of technology Is not irrefevant to my distussioh'of inconipreiensibiity. _ 


"| belfeve that there Is a correspondenée betwen our trust in computer sys- 
tems and the purposes of the systems (béth our understanding of these purposes 
and how weil the systems actually fulffi théi);: ‘that’! there ‘Is a discrepancy 
between the real and the perceived purposes ‘of méity compiter:systema,-and that 
there Is a strong relationship between: our comprehension of these: purposes. and 
the appropriateness of trusting the systems. It Is our trust in computer systems 
which makes us’ so vulnerable to their effects. 

One direct and Sangercts effect of pusting Incomprehensibie. everene is the 
erosion of the. meaning. of. Fesponelbility for one’s actions, . “Who. is to be held 
responsible for modern computer systems: that: neve ‘literally evolved. into their 


present forms and that simply cannot be sald to have authors? Already, some 
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people are suggesting that the computer be responsible for its share of the dect 
sion making toad, but it Is. far from clear what possible meaning computer responsi 
bility could have for the people affected by machine made decisions. Some advo 
cate the idea that systems retain contro! over ongoing processes. in SbactiCg: 
incomprehensible systems become autonomous, unchallengeable authorities to whom 
a society of users abdicate responsibility. According to Joseph Welzenbaum, 
* ... responsibility has attogether evaporated. No huthan is any longer responsible 
for ‘what the machine says.’ Thus there can be neither right: nor wrong, no ques- 
tlon of Justice, no theory with which one can agrée or disagree, and finally no basis 
on which one can challenge ‘what the machine says’2® 

Certainly | do not wish to leave the reader with nothing but dismal pro- 
nouncements about the impossibility of individual action against an autonomous 
technology. in Chapter 6, | discuss alternate courses of action In the face of 
incomprehensible computer systems. First, | consider various technical "solutions." 
These include verification proofs, reliability studies, and modern programming prac- 
tices like structured programming. In the course of analyzing these and other 
means of injecting understandability into computer systems, it becomes clear that, 
for the most part, they attack only the incomprehensibility of computer programs. 
However, the enforcement of structure on the product (the program) does not 
necessarily enforce structure or comprehensibllity on the process which created it 
(the design of a system), and it is the larger computer system that is the concern 
of this thesis. 

1 believe that system incomprehensibility is fundamentally not a problem in 


the engineering sense of that word, and that the most interesting kinds of 


S joseph Weizenbaum, "On the Impact of the Computer on Society," Science, 
176 (May 12, 1972), 613. 
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incomprehensibility are not. ‘s be dealt with by technical means. Rather, we. must 
make a conscious effort to. widen our perspective beyond a narrow . professional 
one. The social consequences of using incomprehensible computer. systems have 
direct impacts on the meaning of responsibility, the ethical and moral burdens of 
computer scientists, and the self image of all peopie. it Is necessary to reckon not 
only the technical, but. also the human costs of the technological system; in Lewis 
Mumford’s words, to ask “agt what Is good for. science or. technglogy,.... , but 
what is good for man.....°% 

There are. positive. steps which can be..taken to. avoid the potentially 
dangerous effects of the existence of vital computer systems that are functionally 
incomprehensible. . As. Vice-Admiral H. G, Rickover has pointed gut, a good. beginning 
may be made by reflecting on whether or not everything halled as progress actually 
contributes to happiness (or whatever else the.reader believes sustains human cul 


ture), and remembering that. we alone are. responsible for our technology.® 


Lewis Mumford, “Authoritarian and Democratic Technics," in Technology and 
Culture, ed. by — Kranzberg and William H. Davenport (ew York: Schocken 
Books, 1972), p. 58 


Sug. Rickover, "A Humanistic Technology,” In Technology and Society, ed. 
by Noel deNevers {USA: Addison-Wesley Publishing Company, 1872), pp. 22-23. 
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Chapter 2: Understanding Computer Systems 


There can be no total understanding and -.no absolutely reliable 
test of understanding. 
he, a _ Joseph Weizenbaum . 


. | understand@.. Geers 
ELIZA 


it is acknowledged within the computer science community that the 


fe 


ineonerehonelsinty of computer systems is a major Problem associated with the 
rapid proliferation of large scale computer eoplications: ‘Nevertheless: | incompreherr 


sibility has proved to be a most difficult concept for both computer scientists and 
yea Spee 


social econnets (working bi technical and nontechnical perspectives) to define. 
Computer specialists hint at the issue of incomprehensibiity when they discuss the 


“software problem," which is really « a | whole cctlection of problems that make com- 


eats Cx 


puter prograne: particularly large ones, intractable. There seems to be something . 


Relat gs 


inherent in large computer systems, perhaps it is the complexity of ‘uch systems, 
which fosters incomprehensibilty. A 1acge: ‘computer system . “worked ‘on by so 


malty people over such a long period of fine that there Is finally n no ) group of people . 


. whe can pe seid to be its authors or who ndoatend it in any “useful. sense; “for 


example, well enough to suarantee its reliability. = 
it ‘Is probably pisnwine ‘to inp right into. a "Geclgeien: “of “computer. 
acBr ei 


Incomprehensbity before saying something donut comprehension in the context in 


which we are pat “familiar with it - that of interpersonal communication. Our “feet 
SHRGG set Bs ite ch 


Ings of uiaetetandina, ‘another person and of being understood. by ‘others are e not. 


Tiseaph Wairenbaue, Computer Power and ‘Human Peksons From icdomeal to. 


Calculation (New. York: OW. H. Freeman, 1976), p., 193. " 
im seat ore pyre ee eH 


24 computer program, published in 1966, which makes ‘possiplaceertain kinds 
of natural language conversation between man and computer. . 
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formally learned, but are patead the rests of pn IOn: Some aeeeee of this 
intuitive knowledge seem to be shared among all | odonie by means of unspoken 
channels; communication is by defiiition an act:of sharing.® “The ne which 
make human communication work (to the extent that it oose Work) most often 
remain unstated. The incomprehensibility of computer systems. is such an elusive 
notion because our sense of what ts means to understand - not externalized when 


we change our. frame of reference {rom that of people ‘to that of machines, and 


begin to talk about understanding computers. 


2.1: Interpersonal Understanding 

As a result of our interactions with other People, we arrive at an informal 
definition of understanding, according to which understanding is a function not only 
of words that are spoken (or thoughts that are communicated in other ways), but 
most importantly of the speaker and the listener, who bring something of them- 
selves to siyjaxchenae with another pebaan: Comprehension, <s implicitly defined 
in human relationships, is to a large extent founded ca shared experiences and 
values. Interactions between people nearly always cane some element of faith, 
based on our trust that the other person will “know what I aeane This trust is 
justified only because “all people have some common formate experiences eared 
There Is consequently some basis of understanding hetween any two humans simply 
because they are human."4 We find that our trust is: best rewarded by people who 


see the world as we do, and in particular by those people who have had expert 


Sthe Oxford English Dictionary LOxtord: Oxford Beata: Etest: 1933), 
p. 699. 

4Weizenbaum, Computer Power and Human Reason: From si a to Calvu-. 
tation, pp. 192-188. 
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ences similar to ones that we have had and that were important to us. An under- 
standing between peopie is not something carelessly acknowledged. it is only 
when we feel that someone eise’s interpretation of the world is sympathetic to our 
own that we are ready to believe thet that person: can understand us. 
One of the most important observations ‘that can be made about human 
_ understanding fs that there are insurmountable timits to the level:of communication 
that can be. achieved. 
Since, in the last analysis, each of our lives is unique, there is 
a limit to what we can bring another person to understand. 
There is an ultimate privacy about each of us that:-ahbsolutely . 
precludes full communication of any of our ideas to the universe 
‘Outside ourselves: and which thus jedlates each: ene of us -from.. 
every other noatic object in the world . 
~ "Fe Knew with: certainty that: @cpesacn: understood what 
has been said to him is to perceive his entire belief structure 
and that is 6quivaient to sharing: his: entire jifezexparience.~. 
Attempts to communicate with other people are largely:acts of faith, substantiated 
in part, though never entirely, by a common. tanguagea, social background, environ 
ment, experiences, etc. |! wish to stress this point.- the: significance. of a common 
humanity In human understanding - because’ it is -precisely .this common humanity 
that ts not part of cur “relationships” with computers, and. thus. cannot play any role 


in an understanding between man and machine. 


2.2: Comprehending Computers 
An emphasis on the human element of interpersonal communication leads 


directly to the belief that understanding must take ona wey different meaning in 


relation to Interactions Serween: men and- machines than a relation to: interactions — 


between people. Just what it Is that characterizes 1 on difference = not immedF 


Sipid., p. 193. 
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ately apparent, since there are a number of. things that we can mean when we say 
we understand a computer system... The nature of our understanding of computers 
depends on the context in which we make use of them.{whieh may be as a neutral 
storehouse of information or as a decision-maker on which we depend heavily) and 
varies in relation to the many different elements that combine to form a computer 
system (from. computers themselves and computer programs to the human beings 


who interact with them, and even the stated and the percelved.goais of a system). 


2.2.1: Program estou’ 

The simplest interpretation of undetstented: Jn: the context of computers is 
that of understanding the: output of a. computer. One. meane of achieving this level 
of comprehension would. be tracing the epenations of.a ameter program(s) that 
generates particular output that we. are interested in and wish to understand. 
Unfortunatety, there are strong pressures that can: effectively praciude such direct 
involvement in computer applications. 

There are numerous social factors which subtly encourage the use of conm- 
puters and discourage any serious criticism of them. One aspect. of. the “American 
way" of life Is the use of the latest technologieal "breakthroughs"; Norbert Wiener 
gives a compelling discussion of what he terms "gadget worshiping": 

Of the devoted priests of power, there are many who f£eagard 
with impatience the limitations of mankind, and in particular the 


limitation. consisting in man’s undependebility- and unpredictea- 
bility.... 

' : (ft addition to the motive which: the gadget worshiper finds 
for his admiration of the machine in its freedom from the human 

‘ Hmitations of speed and accuracy, there-ia.ane-motive which it 
is harder to establish In any concrete case, but which must play 

- @ Very considerable role neverthelnss. .It is the. desire to avoid 
the personal responsibility for a dangerous or disastrous deck 
sion by placing the responsibility elsewhere: on chance, on 
human superiors and thelr policies which one cannot question, or 
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on a mechanical device which one cennot fully understand but 
which has a presumed objectivity . .. . 
Once such a master becomes aware that some of the sup- 

posedily human functions of his slaves may be transferred to 

machines, fe is delighted: At last he has found. the new 

subordinate - efficient, subservient, dependable in his action, 

never talking back, swift; and not demanding a single thought of 

personal consideration. 
Today, gadget worshiping is often implicit in the use of large computerized informa- 
tion systems. In the minds of many people and perhaps of society « as a whole, an 
unwillingness | to make use of current technology bespeaks a backwards ‘attitude 
and even a lack of support for good old American Ingenuity. People who: suggest 
checking up on computers (which surprisingly many people stiil think of as the 
machines that never make mistakes) are thought of as simply not keeping up with 
the times, and because of this they may not be deemed suitable for professional 
advancement. Few peopie can be expected to stand up to the social pressures 
exerted by “ workniace (in the form of professional recognition) and by a society 
in which the omnipresence of ieommoray. is almost Somprerery, accented: 

Wholly apart from the ways in which a pomputor system may directly and 
tangibly benefit a company, computerizing one’s business operation Is a status 
enhancing act; computers are image buliders. It is not “aitmcutt to find serious busk 
nessmen (and even more frighteningly, serious computer science researchers) who 
use computers at least to some extent simply because everyone else uses them, 
even though this justification for employing computers reduces to nothing more than 
the age old syndrome of “keeping up with the Joneses." In a recent public talk, a 
bank executive aceitted that some of the postrone counselors In his department 


display - cumbuter output primarily to impress. tents by enhancing the s Rees of the 


SNorbert Wiener, God and Golem, inc. (Cambridge, MA: The MIT Press, 1964), 
pp. 53-55. 
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bank.” In some cases, it is not much more.-than. such insubstantial reasons that 
underlies the large investments of time and money needed. to build. a computerized 
business system. Given the predominantly social role that some systems fulfill, it is 
not surprising that understandability merits a low. priartty in these. situations 

In addition to social considerations, there are strong economic factors that 
have a suiee influence on a business's use of computers. The pressures ‘of 
economic comrevinn can create a situation that Is far more dangerous than mere 
wastefulness suggests. The real danger lies in the fact that constraints of ‘time 
and money prevent most users of Computer’ generated output from verifying the 
correctness of the output that they must often depend upon in making decisions. 
John Kemeny, president of Dartmouth College, remarks that “It is a simple economic 
calculation that a man who earns $26,000 ‘a Veer caiiot ater to spend a week 
doing by hand something that a computer can do In five minutes."® Unfortunately, 
the opposite may be true. Particularly in circumstances where vital decisions are 
being made (e.g., military command and control applications), what we cannot afford 
is an unjustified dependance on information and decisions output by a computer. 
Nevertheless, given the existing emphasis on cost-effectiveness in the business 
world, understanding computer systems by directiy monitoring their output is fre- 


quently not feasible. 


7Laurence Reineman (Vice-President, First National Bank, Boston, Mas- 
sachusetts), "Computers and the Workptace,” lecture sponsered by: MIT-Program: for 
Science, Technology, and Society, March 7, 1979. 


8 john Kemeny, Man and the Computer — Yer: anes Serbner's Sons, 
1972), p. 107. 
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2.2.2: Theory of Behavior 

A more realistic way of satisfying ourselves that we understand a computer 
system (and the same way in which psychologists often satisfy themselves that 
they understand some aspect of human beings) ls to have a theory of the behavior 
of the system, to which we can turn in order to explain and to verify the output of 
the system. Such a theory can prove extremely useful in understanding the opera- 
tion of a computer system and not just the individual programs that comprise it. 
This higher level of explanation is more. economical than a detailed account of com- 
puter programs (perhaps in the form of commented program code or lengthy docu- 
mentation), and It allows us to predict the output of @ program that Is the jmplemen- 
tation of the theory and to establish when thet program malfunctions. It will still be 
possible for a sufficiently complex system to surprise us, but this need not invali- 
date our theory; rather, nomstandard system behavior may act asa test of the 
vaildity of a theory, and the theory itself can serve as a check of the correctness 
of the system. 

There are certain trade-offs exacted In return for the convenience and the 
security of having a well founded theoretical! explanation of a computer system. In 
defining our understanding of a computer system by our understanding of the theory 
behind the system, we are enforcing soma distance between the system and its 
users. People who make use of a computer: system may come to think of It ‘more In 
terms of the theory that constitutes a behavioral abstraction of the system than of 
the underlying hardware and software. Tite, emeue the existence of a theorett 
cal foundation may make clear when a ‘computer system des something wrong, ft 
will require a shift in perspective to know ‘how to modity: the “underlying “code to 
prevent future occurrences of the aberrant behavior ‘of the system. The code 


itself may be largely incomprehensible, so that modifications may very well damaye 
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system reliability. In any case where a system "bug"? ia iscovered, but elther its 
source persists in being elusive or the costs of modifying’the system are too great, 
users of the system are left with a choice of abandonirig the system or adapting 
their behavior to the system's quirks. Alf too often, people ‘come to depend on a 
computer system too much to consider working without’ it and reverting to older, 
pre-computer ways of doing things. Thus, it Is not ufusual ‘to see af adjustment of 
human needs so that they may better correspond to that with which existing com 
puter systems can reliably (which Is to say, without much Intervention on our part) 
provide us. 

When a computer system is described In terms of a ‘theory of its use, the 
problem of comprehending the system can be viewed as a collection of problems 
that involves understanding first the théory Itself arid thén the relationship 
between the theory and the system.'© The most easily comprehended systems are 
those that deal with some widely understood, unambiguous subject matter. An 
example of a system that is extremely large and complex, but nevertheless under- 
standable in these terms, is the MACSYMA computer system for solving problems in 
the integral calculus. The mathematical theory on which this system is based is 
sufficiently well defined that it is not open to much dispute nor subject to major 


modifications. If MACSYMA makes a “mistake,” the user should be able to detect 


San unwanted and unintended property of a program. 


101, other words, the system is a mode! of the theory: " ... if we view the 
theory we Incorporate into a program as an. uninterprated theory, ‘we are free to 
view the computers behavior as satisfying one interpretation of the theory, so that 
any programmed. computer.can be viewed as ing ting. (an one Interpretation) 
the theory Incorporated in its program. The tokens of computer behavior that on 
ona interpretation are uttered descriptions of the hehavig, of some other entity In- 
stantiating the theory, can on another interpretation be viewed as a ir 
stances of . behavior predicted . by ...the.. theory." .Daniel . . Dennett, 
Brainstorms - Philosophicel aoe on Mind and Paychology, (USA: ti Sdeded Books, 
1978), p. 194n. 
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the error by referring to this underlying theory. Computer system users need some 
knowledge of the theory according to which a system has been constructed in 
order to avoid a situation where a system is set up as an unquestionable authority 
and to justify the trust that is implicit in many interactions between computers and 
people. 

Once we have accepted the validity of a theory, there is another aspect of 
understanding a computer system: we must convince ourselves that the system is 
a valid implementation of the theory. Ideally (if our understanding of the theory is 
nearly complete), we can subject a system to a number of samele cases designed 
to exhaustively test it. If a system fails to crack under the stress of extensive, 
worst-case testing (assuming we can always implement such testing), our 
confidence in it will be at least partially established. None of this, however, 
touches the question of when is it appropriate for us to rely on a Samputar system. 
The appropriateness of trusting a system in a given application depends on much 
more than even the most thorough testing; |! will have more to say about this issue 
later in this paper. 

The reason most often given to justify the need for computers is the com- 
plexity of present day society - the immense quantity of information that must be 
processed and the variety of connections between data that must be stored and 
analyzed. If a theory captures the essential means and ends of a problem we set 
out to solve with the aid of computers, then expanding our view of a computer sys- 
tem from the level of its actual components (such as physical devices and 
software) to that of the theory on which it is based can be a great aid in coping 
with the complexity of many modern systems. Enormously large, complicated sys- 
tems like MACSYMA deal with complexity by giving up local understanding of the 


details of the system, while maintaining a more global understanding of the goals of 
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the system and how it goes about achieving them. 

We may not be able = cuarentos high level understanding of a system if It 
is not founded on robust theories; however, ‘the existence of a well founded 
“theory of use" can enable us to comprehend a large computer system without hav 
Ing to keep track of the numerous details of its operation. Most of MACSYMA's 
users can sunderetend the system well enough for their cpurpcese without being 
aware of the sottware that interprets and cartes out their. requests. We may 
relinquish direct understanding of ara d details of a ‘computer system to ecconener 
very large or “complex tasks that are better understood ‘in terms « of a concise 
theoretical expranation than in terms of a series computer programs. Some com- 
pirter scptications are simply too large 5: scope ‘to be usetmy understoo? in detail 
by anyone. It is important to note thet the difficulties raised by the complexity of 
such applications are surmounted more by our own “understanding of the problem 
domain (which may be In the form of a , well constricted theory) than by the partic- 


ular computer system that may implement a ‘solution to the problem. 


2.2.3: Criteria and Context of Use 

Our understanding of any computer systom depends on ) the context In which 
we make use of it and the criteria according to which we judge it. We may 
comprehend a particular system well enough to feel confident of its a in 
normal usage, but not well encuale'’ to risk making Grenoe? to it. It : seems reason . 
able to expect the ecepiter systems we use to demonstrate high standards of 
reliability, maintainability, flexibility, modifiabllity, etc. Many systems satisfy some 
of these criteria, but it Is questionable how many adatoms achieve enough of them 
to really satisfy their users or to justify thelr continued use. 


It Is the job of system designers to decide which characteristics will be 


26. . Chapter 2: Understanding Computer Systems 


most vital to the operation of a given system, and to ‘make clear the extent to 
which certain criteria will be satisfied by the system. It Is then up to a system’s 
users to be aware of the strengths and weakness of the system (just as we 
accept limitations in people), and not to attempt to Interact with the system in 
ways that are contrary to its "character." In other words, people who deal in any 
context with computer systems should consciously decide in: what ways it Is 
appropriate for them to interact with a computer. For instance, It seems obvious 
that a system that is not designed to reliably support modifications (such as the 
federal social security computer system) should probably not be modified once it is 
in operation. 

. Unfortunately, , simplistic guidelines like the one mentioned above are not 
likely to be of much use ‘in the complicated environment that surrounds a large com- 
puter system. To begin with, modifiability and other Important properties of any 
computer system are in general not "built Into" a system In the process of system 
design or implementation.’* in practice, the major Influences on the design of a 
computer system often turn out to be the cost of the system and externally 
Imposed deadiines for its completion. Computer professionals must at times settle 
for getting a system up and running with the smallest respectable amount of test- 
ing and documentation. Attributes like reliability may not ‘ba investigated until a sys- 
tenn Is re us@ and: * oven a cies, ‘until pecpia ‘fiave ‘pega to” “depoaa on It; “ait-this : 
point, it may prove necessary to make changes to a system whose rellability In the 
face of modifications is highly questionable. 


It is often the case that the extent to which a computer system exhibits 
1 Tan important exception to this statement is the following: programmers 
and system designers often ieave "hooks" in: thelr programs; so that foreseen ex- 


tensions can be made more easily and more reliably in the future. However, one’s 
vision of what hooks should be left Is often severely. limited.: 
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ree gine 


and that our understanding of a system Is formed by means of experiments" ever 
the course of time. Since the consequences of computer systems: can be both norr 
trivial and irreversible, it is of the uyncst importance for us Las “consider oho are the 
real experimental animals. Clearly, it Is not always possible or advisable to deter- 
mine a system's correctness or resistance to change by, for instance, Implementing 
some crucial modifications and. watching the results of the changes on the actual 
operation of the system (as opposed to a test run). | ‘ 
The program code of commercial systems can | easily develop nts a patch 
work of quick fixes for unforeseen difficulties. Clever but obscure technical 
patches may be hastily applied to the original programs in response to pressures 
from corporate management for minimal Production tne and system cost, and from 
users for minimal system denn aie Tracking down a difficult “bug” can lead ‘to 
insights which, if perceived before the _ System was implemented, would have 
resulted in a different system organization than the existing one. However, once a 
system is in operation (in fact, once Its implementation is under way), restructuring 
it. according to new knowledge Is eualy not feasible. Besides ‘the obvious 
economic factors which effectively preclude serious perpeniceton mee are 
psychological influences: after the interesting, challenging work of locating a bug Is 
finished, the job of restructuring a system may appear comparatively tedious. 
Thus, even If there Is no pressure to work quickly, the semetacen to patch up a 
system, rather than to attempt to. restructure Rt, is strong. “Unfortunately, it is fre- 
quently impossible to predict the behavior of a computer system. that consists of a 
collection of patched-together programs. 
| Particularly in the. case of systems with uaclear va potentially dangerous, 


nonreversible consequences, issues like reliability. should. be analyzed well before 
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the system Is put together. Such an analysis should be made from a nortechnical 
viewpoint as well as from the standard technical viewpoint according to which some 
computer systems are already judged. A given reliability factor may warrant the 
use of one system in an ‘academic application, but not the use of another system 
as an alr traffic controller, and the methods of making these Judgments are not 
exclusively those of a formal technical study. 

The most thorough analysis of a computer system would have to take into 
account subtle Issues, such as the degree to which users of a system are likely to 
depend on It and the possible consequences of this dependence. Issues Ilke these, 
which bring Into play the people who use, depend on, and ¢@re otherwise affected 
by computers, are likely to be wate inscrutable than computer systems themselves. 
Although there has been recent evidence within academic. circles of sensitivity to 
the "human factors" of computer systems, few critical, non-technical sentiments 
have filtered through enough economic and social channels to be evident in the 
design and use of existing computer sy stoms. 

Our knowledge of the extent to which a computer system measures up to 
certain important standards (which will in all probability be tested at some point in 
the course of its lifetime) Is a major factor in our understanding of that system. A 
lack of attention to criteria such as those | have talked about here will severely 
limit the ways in which we can reliably, comprehensibly interact with computer sys- 
tems. Shallow levels of understanding are _hardly enough to warrant the high 
degree of trust that many people place in computers. The advisability of exercls- 
ing great caution in constructing and using maser and better computer. systems 
seems. obvious (particularly in. the - unexampled,. poorly vadestaad areas of some 
current computer sclence research), but some of the most highly respected .conr 


puter scientists proceed in their work largely by employment of the method of trial 
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and error, without much consideration of the ramifications of thelr work. 

As a result of the previous discussion, wa Can see that the criteria by which 
we judge a computer system are closely related to the context In which we use it. 
Different, and probably less stringent cfiteria would be appropriate In the considere- 
tion of an airline reservation system than of a military command and control system. 
Context and criteria of use help determine ‘the risks “and benefits of using a com- 
puter system - what might be called the system's “degree of vitality" = on which 
any discussion of the appropriateness of utilizing ‘thé systein should be'based. Ult 
mately, a Computer system's degree of vitaltty depends | on the” ‘ways in which the 
system may affect people. ae es = 

. Finally, the. context of our use of a - computer system tias something to do 
with our perception of the relationship between the system*and the problem It pur- 
ports to solve - the extent to which a computer system attacks the problem we 
actually want it to solve. As | will discuss later in this thesis, computer systems 
designed to solve social "problems" (for example, the failure of most modern school 
systems to educate or to motivate their students) often “miss their mark. "Many 
times, this is so not because of poor programming, but because we do not under- 
stand the nature of the problem.’ In applying computers’ ta “‘Felatively superficial 
symptom (such as the current shortage of teachers) of a persistent social Sidblem, 
rather than coming to grips with the real source of the difficulty (which may or may 
not be "computable"? 2), we are taking the easy way out of the problem. We may 
understand how our system deals with the problem symptoms on which we have 
concentrated our attention, but we wi pay et pexsunecnca sere 

‘W2py “computable,” ! do not mean to refer to any technical definition of 


what can and what cannot be represented bya ‘computer algorithm, but rather ‘to « 
more Intuitive sense of the ee of problems to which a computer system may ap- 


proprietely be apptied. 
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does not solve the original problem (and it Is likely that it will not solve it), since 
we never acknowledged the root of the problem - the ‘human sources of the 
difficulty - in the first place. The utilization ‘of ‘computers ‘as "patch" solutions to 
poorly understood problems can be the source of a subtle kind of Incomprehensibit- 4 
ity that is extremely difficult to detect. The incomprehensilty of such systems 
derives from the absence a a well thought out explanation of the “difficulty at 
hand, preceding the design « of a imputed system to resolve ‘that ‘difficulty. When a 
system is constructed to mode! or otherwise refer to. a human activity, its design 
should refiect first and foremost an ee ee of that activity and. not Just an 


attempt to operationally popmeetet 


2.2.4: Front End Versus Back End 

in discussing computer systems, we iat make clear whether our viewpoint 
is that Ls the designers of a Syaten or Hot tte users. Understanding takes on a 
different maeiing with respect to the design ‘and generation ‘ofa computer system 
(what | will refer to as the "front end* of the eyatem) then it does with respect to 
the use of a system (the “back end"). . | | 

A system’s front end comes into being before the system is even bullt, with 
the basic decision to use computers to deal with a particular problem. Before the . 
technical design of the system can begin, an understanding of the problem must be 
arrived at. Often, the people anes pave the most complete knowledge of the prob- 
lem domain are not the same people who set up a ccmputer system in that domain. 
Thus, in addition to difficulties inherent In the problem being analyzed, there is likely 
to be another stumbling block - that of Communication between the technical spe- 
clalists who set up the computer system and the non-technical workers ‘who provide 


input into the abedincadone of the system aie who will Sccntiany make use of it. 
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An obvious way of seems: with this pomeuvcation ene is to include 
both computer engineers and system users in the design of a svstens “Unfor- 
tunately, there is often considerable difficulty o bridging the gap between different 
perspectives of a computer system (that of the engineers ‘and that of the users, 
for instance). This difficulty oon necessarily be resolved technically (tor  exenr 
ple, by developing a sufficiently high Javea programming language that serves the 
needs. ad me various groups of pecrs who use a 2 computer ‘eyaton): nice: it ‘ anaas, 
at least in Part, vom discrepancies in the ways in which diferent ¢ people interact 
with and iba about computers. . . , 

Besides difficuities with éomemunication, there are other difficulties that ‘can 
preciude a well organized front end of a system. ‘onedat pig eae that Cosbitare: 
are often introduced into situations which have ede peer overtly erganzee along 

“er paee os 

rational lines. bidied the suggestion of rae drones operations, an existing, informal 
system must be converted Into a form that is sulteble for representation by con 
puter programs. in applications outside the realm of f academic veaesice (.e., apple 
cations that are often characterized to a larger extent: by "sloppy" numen interac- 
tions than by well defined ness the task of rationalizing the work often proceeds 
in a haphazard, ad hoc fashion. Moreover, it is often the case that the ‘work itself 
is redefined In terms of the tools in the present discussion, , computers) that are 
available. 

One case study which exemplifies the diticuity of converting areca puter 
activities Into compuerized systems Is ii design ans implementation of the Bank 
of America computer aveten In the late 1960" The designers of this system 
quickly discovered that there was no organized eveten of rules and regulations 
that completely governed the acuity of banking. | Despite the large scale and the 


‘obvious complexity of this system, transactions that were not strictly routine were 
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_ often handied informally, many on a case-by-case basis. In this way, bank persor 


nel were able to take into account unusual aspects of a client’s situation and, to 


some extent, personalize their decisions. As the banking system grew to large pro. 


portions, no formal policy was ever necessary because the system worked. 
Because life gives feedback, contradictions can be resoived informally (sometimes 
we call this leniency). | 

It may not always be in the best interests of a system to formalize its dect- 
sion making processes. in the case of the Bank of America system, the decision to 
utilize a computer system necessitated the sstebsenment ofa system of rules that 
explained banking procecures; as a result, the banking system was changed. -Per- 
sonalized decision making was discouraged as a result of the introduction of compu 
terized banking. Contrary to Vice-Admiral H. G. Rickover’s claim that "of technology 
it can be truly sald that It is not ‘either good oe bad: but thinking makes It so7#18 
the computer did not remain neutral with eecoade to the activity of banking. The 
presence of the tool substantively changed the eee that was computerized, 
even though this was not the original intention. 

Haphazardness in the organization of the ‘atront end of a computer system is 
bound to propagate to the back end, as a result, a system’s users will experience 
varying degrees of difficulty and confusion in utilizing the syston. When a com 
puter system is introduced to replace an activity that waked significant use of non 
standardized channels of communication, ienvetant characteristics of that activity 
are liable to be lost in the process of converting it Into a computable form. Users . 


may find that different decisions are being mada: than had previously been the - 


18Vigg-admiral H. G. Rickover, "A Humanistic Technology," in Technology and 
Society, ed. by Noel de Nevers (USA: Addison-Wesley Publishing Company, 1972), 
p. 21. 
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case, that seemingly small mistakes can have faipibsbidecs: Uiforeseet consequences, 
and that exceptions — are no longer as easy ‘to. handie as ‘they once were. More 
significantly, users may be led -” depend ona computer system that acts according 
to a _patched-together theory of thelr work that Is entirety unciear to them. If 
there Is no well formed theory of use behind a computer system; then the operation : 
of the system is little better than scperinantal: In east canes? dependance ona 


computer system seems highly irresponsible. 


2. 2. 5: Systems 

Before going any further in this discussion, I should sey something about the. 
mpporsant distinction peter Loom Puree programs and computer systems, Much of 
the Incomprehensibilty ¢ on which concerned ‘computer sclentists ond up focusing Is 
that of computer programs. and, to a ‘emate! extent, of the: processes of problem 
analysis, design, and Programming Haelty wach cumnate in the generation. of ‘pro 
gram code. We are exerclone oxtremely | narrow ; vision i restricting our view of 
computer tam comnranedaninty to the incomprehensblty of one part (abelt 
the most logical and well understood 1 part) of computer systems - computer pro- 
grams. . 
Butler Lampson, a sont research fellow of the Xerox Corporation, has 
stated his belief that "it Is the source text that completely defines the [computer] 
system. 14 it is not t surprising that his ideas about making computer systems more 
comprehensible inven s Schemes for things like bulaing structure Into program code 
(through greater sitentica to program hierarchy. and ‘nterfaces between programs). 


Lampson claims that by making explicit all changes. to: onde (for instance, by 


- 14gatter Lampson, “Building Programs,” Mit: pence for. raises! Sclence 
Distinguished Lecturer ‘Series, May 1, 279. fue 4 
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keeping track of the editor in some well defined way), we will be able to explicitly 
capture "the nature of the changes," and thus figure out what is going on. 
Lampson’s concerns about the dynamic nature of program evolution are valid ones. 
However, his persistence In keeping everything but computer programs themselves 
out of his treatment of incomprehensibility ignores the fact that a complete system 
Is more than Just computers and the other machinery associated with thelr opera- 
tion. 

Even assuming that there existed unambiguous, organized methods of making 
_Gomputer programs understandable (and this in itself presupposes that we know 
and agree upon what it means to understand a computer program), a computer sys- 
tem cannot be rendered comprehensible simply by “bullding" understandabillty into 
its software. For instance, | have already mentioned the importance of taking into 
account the context of use of a computer system in any discussion of the system. 
.-.. most large systems actually in use today and on which 
people depend... simply are not logically deterministic sys- 
tems in any useful sense of the term. It no longer makes sense 
to speak of such systems as having a state at a particular 
time, or of their programs as if these were concrete texts hav- 
ing an existence independent of the detailed circumstances in 

which they are used. 

T. D. Sterling, writing in a recent article on the social impacts of computing, 
expresses my own desire for a wide view of computer systems; he repeats Kling 
and Scacchis definition of a computerized “package”: “not only devices (e.g., 
hardware, software, and systems protocols), but also a diverse set of skills, organi 


zational units to supply and maintain computer based services and data, and sets 


of belief of what computing is good for and how it may be used efficaciously." 15 


15 joseph Weizenbaum, "Human Choice in the Interstices of the 
Megamachine;" pp. 12-13. Lecture’ presented at the IFIPS Conference: on “Human 
Choice and Computers," Vienna, Austria, in June, 1979. 


16,, D. Sterling, "Consumer Difficulties with Computerized Transactions: An 
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Like these people, by referring to computer systems | mean to suggest something 
with many dimensions, some of which have found no place in purely technical dis- 
cussions of computer incomprehensibility. The ‘aiFencompassing nature of computer 
systems is what makes It so difficult to define and to deal with thé Incomprehenst 
bility that derives from them, but the frequent tendency to shy away from con- 
sideration of the societal aspects of computers has helpéd entrench the 


Faas Serr a 


incomprehensibility of some systems. _ 


Emptirical Investigation," Communications of the ACM, 22 (May, 1978), 284. 
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Chapter 3: Sources of Incomprehensibility 


Evidently technological accomptishment has become a tempta- 
tion that no person can reasonably be expected to resist. The 
fact that something is technically sweet is ensegh to werrant 
placing the world in jeopardy. 
Langdon Winner 
It is hard to realistically deny the pervasiveness of incomprehensible com 
puter systems In modern society. Most of the computer ‘selentists. with whom i 
have spoken have ready examples of thelr pet: incomprehensible systems (large 
operating systems are favorite choices). In discussing incomprehensibility, many of 
these people smile knowingly at me as they remerk that it certainly isn’t. difficult to 
find instances of baffing computer systems: What | find ‘particularly upsetting 
about these conversations is the fact that the comments:are made from a profes- 
sional viewpoint; that is, the systems to which ‘these peepie- refer are felt to be 
incomprehensible to computer specialists. What, then, is: the position of all the 
non-technical people who make use of and are otherwise affected by large sys- 
tems; how are these people to understand a computer system. that the “experts” 
have called incomprehensible? If the seriousness ‘of the probtem. of incomprehenst- 
bility is acknowledged by computer scientists, why ts it that incomprehensible sys- 
tems are so widely used today and that aaity systems currently being constructed 
are likely to emerge lidomprenenebie in some sense. of the word? 
in this chapter, | consider the question of howe does Incomprehensibility arise 
in a computer system. Like most questions: that involve ‘human interactions and 
needs (and those dealing with computer systems do), this one does not have a sin 


gle, well defined answer. Rather, computer system incomprehensibility is a problem 


TLangdon Winner, Autonomous Technology (Cambridge, MA: The MIT Press, 
1977), p. 73. 
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me BES. - 


that brings together a host of different Issues. The “reasons" that we may come 
up with to explain the existence of i emesenedsole cyetens dedend on our point 
of view - that of @ researcher, programmer, designer, businessman or other user, 
social critic, etc. For-the-purposes of this discussion, Incomprehensibility Is best 


considered from all these perspectives. 


3.1: The Programming Process 
Computer programs are among the most ebvious components of computer 
systems, so we may begin an. analysis of the sources.of incomprehensibility at the 
level of computer programming, | have previously discugsad some problems of conr 
munication that. beset many larga programming projects... Most software workers 
are aware that there are a variety of: difficulties with virtually every stage of the 
construction of a computer system, and that these difficulties confound program 
mers working both individually and as part of a.larger endeavor. | 
On a personal level, programmers are.viewed. by.much of society as rather 
pecultar people. Computer scientists must.cope with unproductive stereotypes that 
emphasize what might be: called the “hacking mentality." 
. the hacker is “without defigite purpose": he cannot set 
before himself a clearly defined long term goal and a pian for 
achieving it, for--he Aas.-only teahnique, not. ige....He_ has 
nothing he can analyze or synthesize; in ‘short, he has ‘nothing 
to form theories about. His .aklll.: ls: .th _abnless,. even 


disembodied. It is simply not connected with apything other 
than the instrument on which jt may. be-exercised.” 


Programmers are sometimes percelyed as having-little purpose other than to spend 
time with computers; they may be seen as solitary. individuals whose strong attact- 


ment to machines is proportional to their detachment.from and lack of experience 


2 Joseph Weizenbaum, Computer Power and remap Reason: From Judgment to 
Calculation (New York: W. H. Freeman, 1976), p-.118.. ; 
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with other people. 

Professional Isolationism Is encouraged oe the training of software work- 
ers, which single rancemy euphasives technical skills. “anadon Winner notes that 
eiiaiioave typically regard themselves as “merely problem solvers": 

“Teil us’ the problem," they demand: “We wilt find a “ g0kston: 

That’s our job. But you may not presume to. question: the nature 

of our solution. You are not a member of a technical profession 

and, therefore, know nothing of relewance. if yoy incist.an rale- 

ing questions about the appropriateness of the means a dev- 

ise, Ce Oy Conus Sa Fee eee aeeeeS: 
The personal identification of programmers with their programs, which occurs in 
many existing set-ups, does not support productive communication between work- 
ers. in addition, workers. in technological fields: are subtly, taught. to accept seg 
mentation in the work structure: that surrounds: them.. Programmers are. discouraged 
from asking questions or exploring issues beyond thelr eeaigned tasks; thay are 
“deprived of all but the most narrow. skills: and of an-uaderstanding of how their 
work fits Into the work process asa whote."4 Codera maybe forbidden toa use any 
programming techniques Sut a few rigidly defined ones, so that their program code 
emerges as little more than “a standardized product :made in in a standardized way 
by people who do the same limited tasks over and ever without knowing how they 
fit into a larger undertaking."© 

Even on a more organized level (e.g., that of the management of a software 
project), the activity of programming is often viewed in.an impersonal way. In his 
book, The Psychology of Computer Programming (which is, incidentally, one of the 
most widely read books about computer pregrammers),.. Gereld.. Weinberg comments 


38 Winner, p. 11. 


“php Kraft, Programmers and -Menagers: The Reutinization of Computer 
Programming in the United States (New York: Springer-Verlag; 1977), p. 83. 


Sipid., p. 59. 
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on the widespread lack of attention devoted to the informal, interperaonal (as 
opposed to formal, organizational) structure of sortware proleces and to the sbolal 
(as opposed to Physical) environment that supports the work of programming. The 
fact that Weinberg finds it necessary to stress the human element of f programming 
is indicative of a ngeneral lack of understanding of and concern : for programmers and 
the interac tors between them, that is not uncommon in ‘aunagetal clrcies. Philip 
Kraft, too, has much to way about the artificially maintained atance between pro- 
grammers and managers, and its negative effects..on. the production of computer 
systems.® 

In addition to psychological factors ‘that. play a significant role in the pro- 
gramming process, there are. other, even more tangible pressures that affect the 
quality of computer programs, often by contributing to their inecrutability. The. most 
obvious of these pressures are caused by short range economic constraints on ali 
programming projects. These constraints. discourage the writing of thorough docu- 
mentation or of well commented, understandable code. -{t is a common folklore 
among peopte who are not terribly familiar with computer programming that good 
code is likely ‘to be more expensive to run than code that consists of program 
“hacks,"’ but this notion Is not supported by practical experience. What is cer- 
tainly true is that obscure code is in the long run costtier to maintain than well writ- 
ten code. Unfortunately, the non-monetary costs of using poorly coded programs 
(which are bound to be unreliable and incomprehensible) are not. reckoned into the 
budgets of the projects that produce them. 

Economic requirements translate into time limitations that tend to pressure 

Sx raft, op. cit. 


La vernacular term used to denote quick, clever pieces of work that are 
generally not very well structured or documented. ; 
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software engineers Into accepting inadequate solutions to problems that they are 
asked to solve. Weinberg voices the opinion of everyone who has ever done any 
serious programming when he states that “we must be realistic and acknowledge 
that probably no perfect ‘program was ever written. Every really large and 
significant program has just one more. bug." Nevertheless, adequate time for care- 
ful program design, meticulous coding.” extensive testing, and complete 
documentation - all of which are necessary if we are ‘even to attempt to produce 
high quality, understandable code — is ‘considered a “Wuxury in most ‘non-academic 
programming situations. in ‘general, it is only the short ‘term achievement of a sys- 
tem that runs which is tangibly vaueided: Because the ‘ong ‘term, global view Is 
. not usually taken, important corisiderations ae be overlooked. For instance, once 
a given computer system is declared operational and is put into use, many of the 
programmers who were Involved in making that system run begin working ‘on other 
systems; thus, the people who help jeans and implement ‘a farge system are not 
usually around when. problems with the system arise. ue 

Another factor that contributes to program incomprehensibility Is the gen- 
erally low level of professionalism that characterizes the activity of programming. 
We think of an activity as being professional when the members themselves deter- 
mine the criteria for membership, ghooes who is to be a member and who is not, 
and have the power to remove someone from the protession If he does not adhere 
to certain collectively set technical and ethical standards. The Council of the 
Association for Computing Machinery has adopted a set of guidelines for profes- 
sional conduct in tore: processing, but pai aia programming Is not really 


governed by anything like a code of ethics; the fact. 4s. that..there Is no 


SGerald Weinberg, The Psychology of Computer Programming (New York: 
Van Nostrand Reinhold Company, 1971), p. 19. 
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professional standing for programmers (the ACM primarily serves academics and 


other computer science researchers;” that Is, people who make up a ‘oman portion 
of all people who work with computers). . . | | 
Current writings that deal with the training of programmers. Indicate that 
software workers” are taught not to question the Conetrawts bbl are imposed on 
thelr. work, even though these constraints may be Imposed by people who have had 
few. dealings. with computers; they are taught to stress corporate profitability and 
to Identity this with technicel rationality and efficiency, even though these criteria 
are not sufficient to determine when a given performance level of a program Is met 
and can be depended upon; in short, they are taught everything but * sense of 
professional ethics. Furthermore, they are _ Subjected | = _gomtinuelly changing 
specifications | from computer users; these, ¢ combined with the coming and going of 
people involved in a computer. Project as the design of the system progresses, 
ensure steady modification that fundamentally changes a computer sytem . In 


unpredictable ways. 


3.2: Complexity 

Some advocates of increasing the use of computers would have is believe 
that computers are both necessary and beneficial to a society as complicated as 
our own. They argue, on the one. hand, that inoue much intellectual aids, our 
complex modern society may well fall apart under Its own weight of complexity, u10 


and, on the other hand, that better (/.e., computerized) means for handling and 


SThis can easily be verified by a casual survey of the affiliations of the « au- 
thors of the articles and istters in ACM-publications.. =... 


TORobert Fano, "The Computer Utility and the Community," IEEE International 
Convention Record, Part.12, 32. 
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disseminating information will result in an Increase in intellectual. freedom and in 
diversity of lifestyles. These arguments form the basis for much of present day 
computer use; yet, as | intend to show, they do not stand sip to critical exanitn: 
tion. . | | | | 

The value of the. computer in dealing with Henge complex problems, and par- 
ticularly with human ares Is + highly questionabie. In some cases, the complex- 
ity of computer applications derives not from: the: problem oman, but from the use 
of computers. {it is actually ‘the ease. that some. computer systems ‘generate com 
plexity and render problems. more . eee Vabieeatite: and: unmanageable than they 
were originally. For instance, Edwin Paxson cenemet to Justify, computerized mil- 
tary defense systems by stating that “there ls littie question: that setious systems 
analysis without the computer Is pepOeeierery he goes on to. undermine his own argu- 
ment by unknowingly naming the computer bicaiashi as the source of : the: sompiextty 
with which it is associated: "As computing) pei: has. increased, 80. aie0 has the 
complexity of the anatyses it fosters.*’ 1 [emphasis mine] ‘Similarly, dohn Kemeny 
first praises Dartmouth Colfege’s computertzed: bookkeeping aystem for maintaining 
many thousands of accounts accurately, and: then acknowledges that "It is also a 
. by-product of the system that we have such a vast. complexity of information . 
that human beings cannot cope with It without’ the: aid: of the computer."'@ in the 
light of what these people and others have said, the argument that computers are 
necessitated by the compiexity of modern society is hardly tenabie. 


The alleged benefits of complex computerized information systems are even 


1TEGwin W. Paxson, “Computers and National Security," Chapter 3 in Com- 
puters and the Problems of Society, ed. by Harold Sackman and Harold Borko (New 
Jersey: AFIPS Press, 1972), p. 77. 


12 john G. Kemeny, Man and the Computer ‘(New ‘York: Charles. Scribner's 
Sons, 1972), p. 104. 
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easier to dismiss than the preceding “argument from necessity. 4 pecor ding to one 
view, canputers: by virtue of their information Sroceseind capabilities, are: part of 
the foundation of a new men age. However, It is not aifficult to see that exeting 
computer systems do not srauide any évidencé of the eosin of this” ‘prosperous 
renee, and there is no compelling reason to belleve that twice as many, or even 
ten times as many systems ever will. _ | 


The record of how technology has actually been used in the 
“modern. world:-does act support [thia] supposition. in principle, 
transportation. and communication might have been expected to 
contribute to: a broadening of ‘-human.-axperience and. greater 
tolerance for different customs and beliefs .... The ability to 

- travel with ease on a giebal scale and.:communicate across 
great distances has not led to increased understanding or great 
compassion. There is.no evidence to suggest that international 
tensions have been lowered = as a Sooeequence of increased 
information fiow . 

The problema ‘of human interaction cannot be reduced to 
information flow, nor can the shortcomings ..of.. existing social 
arrangements be attributed to Imperfections In our Instruments 
of communication.. Is there any reasen ta believe. that. a two 
way terminal in the home would materially alter an individual's 
response to television breedcasta showing scenes of. violence 
or human mercy? The mere fact of having access to informa- 
tion doss not create the disposition to .act appropriately... .. 
It is purely wishful thinking to suppose that improved information 

’ flow will result in spentaneous efforts to resolve conflict and 
create more responsive social environments. mia historical evi 
dence points to further conceatration of power." 


Mowshowitz ‘drives home the point that although Information is. necessary for 
rational decision making, it is not sufficient for harmonious social interaction. | 

In discussing complexity, | do not mean.to refer to the formal definitions of 
complexity utilized by computer science researchers, but rather to a more Intuitive 
definition. A complex computer system is one that is composed of many parts that 


are interrelated in complicated ways, so that it may very well be impossibie for us 


13 abnbe Mowshowitz, The Conquest of Will: Information Processing in Human 
Affairs (USA: Addison-Wesley Publishing Company, 1976), pp> 164-65. 
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to keep track of all the systems’ components and the relationships between them. 
In reference to truly complex computer applications, | have already discussed the 
Importance of a well understood theoretical foundation; a computer model (or any 
model, for that matter) is only as good as the theory behind ‘it. The point here Is 
that complexity is not a mandate for incomprehensibility: witness the | MACSYMA 
system (mentioned in Chapter 2) and the DENDRAL system, which implements a 
theory of mass spectrometry. * ... incomprehensibility Is. not a necessary pro- 
perty of even huge computer systems. The secret of their comprehensibility lies in 
that these systems are models of very robust theories." 14 

Most computer systems in use today “don’t deal with complexity at all - nor 
are they designed to - they deal with sheer magnitude." This Is often the case 
with the huge information banks utilized by so many businesses. “In these applica- 
tions, computers have become necessary only In the sense that certain services 
could not be rendered in thelr present form without the computer. However, the 
present form of these functions has been dictated in large part by the aveltebilty 
of and early dependance on ‘Computers. ‘This state of widespread computer use 
and dependance, which now appears inevitable and inescapable, is In reality man 
made. Apparent pomerexity may be deceiving and does not uhuaue warcant the use 
of computers, which may only conceal less technological, "perhaps more perepioe: 
clous ways of solving certain problems. wae 

| have mentioned 1 complexity in my discussion of incomprehensible computer 


systems because It is ¢ one quality that Is nearly always associated with ‘computers 


14 Joseph Welzenbaum, "Human Choice in the interstices of the 
Megamachine,". p. 13. Lectuse presented et.the IFIPS Conference. Ot “Human 
Choice and Computers," Vienna, Austele,..in June, 1979. 


18)p 1, p. 7. 
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and that figures largely in their Incomprshensiehity: Snaiow understanding of a 
complex problem may result in. a system that ls unnecessarily pede. Untor- 
tunately, our assumption that computer aystene: are too compre* Tor the average 
person to understand has helped prevent the exposure of, much existing system 
incomprehensibility. In dealing with computer systems, it is well to consider both the 
easily forgotten fact that Snormously complicated tasks. were success tuny carried 

out before the advent of compute: and ae easily obscured yece that the modem 


megamachine'® has not heralded an age of numen mannnest: 


3.3: Evolutionary Systems 

One of the reasons that even computer sclentists are worried about 
incomprehensible systems is the current reality of ‘computer systems that have 
grown to a point where they are no longer under =e conte: of me people who con- 
ceived and created mem: A large system ls woes on by 80 many: people over 
such a long period of time that by the time it is completed, here is no person or 
small group of people who can be said to understand it. 

One of the most disturbing facts: about large computer systems is mat there 
is ‘no group of people who can be identified as ne authors of the system; in other 
words, no one is ultimately responsible for the operation of the system. Virtually all 
large computer systems are far too difficult to comprehend for anyone to be able to 
modify them _without risking unpredictable consesuencee: certainty this is ateave 
the case with systems of whose evolution. we cannot keep track. ‘Once Say: are 


put into operation, these systems are, for the most er) aun to change - they 


16, term used by Lewis Mumford to deseribe teosive 0 ofganizations intended 
to carry out tasks whose magnitude places ‘them beyond the sia peated of email 
groups of people. 


46. Chapter 3: Sources of Incomprehensibility 


can only grow. 

Herbert Simon, a leading computer specialist, describes a computer program 
as a strategy of action whose states and ‘pute ‘eed not. be envisioned in 
advance by the programmmer. '” Such an attitude sees not support the security 
‘derived from ihe commonly held view that computers do only what they are told to 
do. in a fearfully real sense, Simon is describing systems whose present form arid 
’ mode of operation were neither Slanned not foreseen when the system was 
designed. These are systems that have evolyed into what they ara now, in ways 
that no one has kept track of or really understands. 


When a program grows in power by an evolution of partially 
understood patches and ‘fixes; the programmer begins to lose 
track of Internal details, loses his ability to predict what will 
happen, begins to -hope’ Instead: of ‘Know, and «watches: the 
results as though the program were an individual whose range 
of behavior is uncertain: 

This is already true In some big programs . it will soon 

' be much more acute ... targe ‘heuristic: programs - will be 
developed and modified by several programmers, each testing 
them on different: examptes from diferent consoles ‘and inserting 
advice independently. The program will grow in effectiveness, 
but no one of the programmers wi understand It all{Of course, 
this won't always be ‘succeésful - the-intersotions: might make It 
worse, and ‘ino one ‘might be able ‘to fix: # dgdin!): Now we see 
the real trouble with statements We “it onty dees wore pro- 
grammer ‘told If to do. There ent oo  iheciorcnnca 


Professor Minsky might well have said that ‘share are no individual progamners at 
all. Gerald Weinberg has étepaitned the team programming concept;1% the sur- 


vival of the team throughout all phases of a: computer system, from design through 


17herbert A. Simon, "What Computers Mean for Man and Society," Science, 
105 (March 18, 1977), 1187. 


18arvin Minsky, "Why Programming is a Good Medium for Expressing Poorly 
Understood and Sloppify Formulated tdeas|": in Dea/gr and. Planning, 1, ed. by M. 
Krampen and P. Seeitz (Néw ‘York: Hastitigs House, -1967); p. 121. Quoted in 
Welzenbaum, Computer Power and Human AGEBSN: : ‘From: nent to C€afculatian, 
p. 236. 


18Weinberg, op. cit. 
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maintenance, encourages the belief that there is a group of people who continually 
maintain control over the system. However, the composition of a team ‘changes so 
often that the team that maintains a system and must deal with problems that arise 
in the course of using it Is fundamentally different from the team that made the ink 
tial oe decisions. thus, In addition to the rt that large computer systems 
adapt as they operate, the group of people that is supposedly in control of the 
system continually ‘changes. It may be impossible to rellably predict the final 
results of a B computer system for quite some time after it has haan put into opera- 
tion and after people have begun aepennng on it; it may es be impossible to rel 
ably predict how well a programming team understands | a system ata oiven ‘time. 
Much ‘of the incomprehensibility: that charagtorizon these evolutionary conr 

puter syeteme derives: from the organization of. the. -Programming of the 
system - the highly segmented, hierarchical ‘strveture . of.a large computer 
project - and frem the programmers themselves - the. constantiy changing nature of 
the group of people who create ‘and. then maintain a ‘computer system. | 

Most of: ‘the: routine-and tedium associated. with.the dally tasks 

of producing: programe are.left. to:an_apeny@ous.aumy of. peaple 

who merely..dq whet they are told, understanding.jittie of what 

they do and lese of why they: are-deing it... At Jeast up until 

how, the computer has intensified, not: reduced, the. ing ole 

between those who think and those who do wash ei else. 

[my emphasie}: 

Although such systems may not be smart enough for us to imagine depending 

on them a great deel, Marvin Minsky.. worries that."uafortunately, there are too 


many ways a dumb system with a huge data base can be useful"! Just as one 
20kraft, p. 29. 


in The .Computer Age: A Twenty-Year View, ed..by Michael: L. "Dertouzos and. gue 
Moses (Cambridge, MA: The MIT Press, jn.press).;. Quoted in Joseph Welzenbaum, 
“Once More: The Computer Revolution," in Michael L. Dertouzos and Joel Moses, ed., 


op. cit. 
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cannot talk usefully about a computer program without also talking about the tect- 
nological system in which It is embedded, one cannot talk usefully about a computer 
Program without considering the database on which it draws for information. it is 
often the case that one cannot distinguish very easily between the program and 
the database system. Modern databases are often dynamic, introducing yet 
another source of. unpredictability into a computer system. 

It Is not dimcult to find people who seriously worry whether we can maintain 
control of our computer systems - whether people can keep up with their machines. 
It Is harder, however, to find someone who recognizes the absurdity of needing to 
ask such questions. | 

But we must win our technological race with competing nations 

fst and then do the best we can with the realignment probiem. 

Remember, the readjustment gjoblem is common to ail 

technologically-advancing nations. 
What Thompson euphemistically calis the “realignment problem“ Is really what 
Joseph Weizenbaum describes as “the feeling of powerlessness so ubiquitous 
among Individuals in our society ... the widespread alienation of people from one 
another and from thelr work .. . the perception of ordinary peopie that they are liv- 
ing in the interstices of a gigantic system.*29 Computer systems usually replace 
older, less technological ways of doing things, and once a workplace is organized 
sround computers, they quickly become Indispensable to the functioning of that 
workplace. If no one is willing to assume responsibility for a system, why are we 


willing to trust that system In norrtrivial areas of our lives? 


22Howard Thompson, Joint Man/Machine Decisions (Cleveland, Ohio: Sys- 
tems Procedures Assoc., 1965), p. 67. 


28 Weizenbaum, “Human Choice in the interstices of the Megamachine," p. 1. 
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Of all the things Inappropriate to the man-made environment of 
the modern aga, none ts so inappropriate as man himself. ‘He 
must adapt himself,’ Ellul comments, ‘as though the world were 
new, to a universe for which he was not created.’ 
Langdon Winner 
Quietly and complacently, it was sinking into hanes 3 and 
progress had come to mean the progress, Of the Machine, 
E. M. Forster 
| have already noted that computer programs should not. be considered 
independently of the circumstances In which they are used; similarly, computer sys- 
tems and system incomprehensibility should be examined from a broad perspective 
that emphasizes the technological aoclety in which we live. Daniel Bell writes that 
. technology is not a ‘relfied thing’ or some abstract ‘logical imperative’ but is 
embedded in a social support system, and it is the support system, not the technol 
ogy, that determines its use."* oe 
There are many aspects of society's attitude toward technology that can 
lead directly to incomprehensibility in computer systems. Our technological society 
operates in “performance mode," whereby issues not directly related to the 
economic value of a system are likely. to be. suppressed. Questions such as 
whether or not we need a particular system.or whether a system is safe (in a 


broader sense than that. exemplified by whether a system is harmless to its 


TLangdon Winner, Autonomous Technology (Cambridge, MA: “The MIT Press, 
1977), p. 216. Quoting Jacques Ellul, The Technological Society: trans. by som 
Wilkinson (New York: Alfred A. Knopf, 1964), p. 826.- 


2c. M. Forster, "The Machine Stops," in E. M. Forster, The Eternal Machine 
and Other Stories (New York: Harcourt Brace Jovanovich, 1928), p. 285. Quoted in 
Abbe Mowstiowitz, The Conquest of Will: information Procedsing’ in’ Human Affairs 
(USA: Addison-Wesley Publishing Company, 1976), p. 313.  * 


Spaniel Bell, "Hard Questions and Soft Minds: A Reply to Weizenbaum," 
Chapter 21 in The Computer Age: A Twenty-Year View, ed. by Michael L. Dertouzos 
and Joel Moses (Cambridge, MA: The MIT Press, in press). 
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operators, for example) may be subdued by questions fike “will it work?" 

There is widespread ‘pressure’ against ‘eritics of computer systems, which 
encourages dependance on these aystews: _This often takes the form of peer 
pressure; counter critics cite the: observable fact that “everyone: ¢ else" seems to 
be using computers. 

And so we conclude that on-line decisions have to be made 
rapidly in some “systems because of the ‘advance in technology; 
_ and have to be made ‘rapitly in business to’ meet the compett 
tion created by learning to make those vital, rapid decisions 
elsewhere. 
The above reasons ~ business conection and edvances in technology - are typk 


cal but not extremely compelling. iusuncatons for the oten uncritical mde cevbigh of 


jarge scale computer epplcations ane for the discouragement c of eaicrem: 


4.1: Autonomous Technology = 
The organization of modern society is founded gn oer many y people perceive 

as Technological necessity - - a wigespresd. sense of technological inevitability that 
Langdon. Winner has called “autonomous technology" 

Autonomous technology Is the part of our being that has been 

transferred, transformed, and separated ‘from living needs and 

creative intelligence. Any effort to reclaim this p part of human 

life must at first seem impractical and even ebsurd. 
Several writers have distinguished ‘bétween technology: aid the broader connotae- 
tlons of technique - a Iimitiess way of organizing the world; an al-encompassing 
arrangement of-which human society is but.one segment.. My Intent throughout this 


chapter is to examine the presence of technique in today’s society. Most of us 


Stoward Thompsan, Joint. Man/Machine Decisions. (Cleveland, Ohio: Systems 
Procedures Assoc., 1965), p. 40. 


- Swinner, p. 333. 
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are governed to a larger extent than we consciously acknowledge by a technolog! 
cal mentality, according to which technique assumes the characteristics of an 
irresistible force and not of just a tool: Gur sense of inner directedness has been 
superseded by a sense of what a believe ow highty organized environment 
*wants" us to do or will allow us to do. 

The technological society Is, to a large extent, a dangerously uncritical one. 
We have allowed our virtues to become technical ones; the epirit of the day is that 
of maximum productivity. Technique “clarifies, arranges, and rationalizes; it does in 
the realm of the abstract what the machine did in the domain: of labor."© Technique. 
specifies attitudes that are valid once and for ail. 

Winner exhorts us to consider “Instances in which things -have become 
senselessly or Inappropriately efficient, speedy; rationalized, measured, or techni- 
cally refined."’ | have encountered numerous examples “6¢a thoughtless -accep- 
tance of technological virtues; | will mention: only: a féw of them: Howard Thompson 
{tin Joint Man/Machine. Decisions) implicitly assunies the desirability of complexity 
and competitiveness; onthe basis of this assumption, he does not bother to justify 
the use of computer systems In his discussion of decision making.© The ‘title. of 
Sheridan and Ferreif's book, Man-Machine Systems: tafermation, Control, and Deci- 
sion Models of Human Performance, would seem ‘to indicate an emphasis on the 
human element of man-machine interactions. However, the ‘authors inform us that 
their consideration of human performance is. necessarily limited to that which their 


models can describe: 


6 jacques Ellul, The Technological Society (New York: Vintage Books, 1964), 
p. 6. sy 


7 winner, p. 230. 
8thompson, op. cit. 


Section 4.1: Autonomous Technology 53. 


People may show grace, imagination, creativity, or feeling even 
in narrowly constrained tasks; but these--quelities are too fine 
for the nets we cast in modeling and experiment. We have to 
‘be content to desgribe and prediat at a.mugh more. mundane - 
level. Our frequent use of terms such as operator and perfor- 
mance instead of persea.or. behavior .js..meent, to. emphasize. the 
_ engineering context and the re mt ee narrow range of human 
. experience which it encompasses. : 
Sheridan and Ferrell certainty deserve credit for recagnizing and bringing to our 
attention some of the distinctions between the human. and the mechanical elements 
of the interactions that they oonsider. Still,-when we. are led to believe that we 
must be content with a narrow range of human experience because technique 
requires this limitation, then. our lives are. being viewed too much in the "“engineer- 
ing context." "... to dwell on these.impressive, statistics which tell us what peo 
ple do, without.attention to how. they. fee! abaut. what they do is to miss a pro- 
foundly important-dimension..of human.experience: - that js, the meaning that peo 
ple attribute to their behavior," 19. ee 
in one article, Harvey. Wheeler, attempts to explain. why we. should use conr 
puters to aid us in. making decisions; .He: states that .we,do not expect perfection 
from either men or machines, but that. all we want is a way to make decisions more 
systematically. ! 1. Again, 1 caf attention to. the; unquestioned assumptions — in this 
ase, that systematicity as such. simply and obviqualy justifies Ktself and is always 
to be preferred over less formal. criteria... Why not .go bayond the question of 
whether or not we can systematize. the way.ia which we make decisions and ask 
“Stnomas B. Sheridan and William R. Ferrell, Man-Machine Systems: Intorme- 


tion, Control, and Decision Models of Humen Performance (Cambridge, MA: The MIT 
press, 1974). 


101 wian Breslow Rubin, Worlds of Pain/Lite in the Working-Class Family 
(New York: Basic Books, inc., 1978), p. 136°” 


1 THarvey Wheeler, “Artificial Reasoning Machines and Society," Chapter 13 
in Computers and the Problems of Society, ed. by Harold Sacknian and: Harold Borko 
(new Jersey: AFIPS Press, 1972), p. 489. ¥ 
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whether or not we really want or ought to do so? 
The use of computers is sometimes justified by the argument that at least 
. computers are iss worse than people (e.g., automated decision making can hardly be 
worse than decision making by executives in a complex bureaucracy), and that the 
computers rationality frequently makes: them preferable to people in seme applica- 
tion 2 often, there Is evidence of a pre-determined approvel of the use of com 
puter technology that may cause us to ignore many undesirable aspects of applica- 
tions. that computer systems efficiently. cover up - aspects that cannot be 
effectively dealt with by a computer system. For example, the ELIZA pbeiuter pro- 
gram. (mentioned in Chapter 2) was hailed by some people as the precursor of a 
psychotherapy machine that would deai with much of the neuroticism of modern 
society better than human therapists are able to do. People who made this judg- 
ment tended to lose sight of the nature of human psychological probleme amidst 
their raptures over the computer. "ELIZA had less to do with showing how much a 
computer can do than with revealing how. cognitively and emotionally empty some 
forms of human interaction can be... wt8 
in our society, there is a strong temptation to submerge one’s own autonomy 
in the megamachine; to give way to "a conscious and unorecous response to 
whatever situation atigee: This response strongly and automatically tepulses any 
alternative mode of action ... it neutralizes alternatives by making them seem 


unnatural, impractical, or simply impossible." 14 in minimizing the role of human 


125everal writers have argued that, although. computer systens may. not 
perform as well as superior human beings, the systems are better than most peo- 
ple. See Kenneth Colby, "Computer Psychotherapists," UCLA Department of Psychi- 
atry, Algorithmic Laboratory of Higher Mental Functions Memo ALHMF-1 4. 


13theodore Roszak, “The Computer - A Little Lower ‘Than the Angels, " The 
Nation, 222 (May 1, 1976), 634. Review of Computer’ ‘Power and Human Reason: 
From Judgment to. Calculation, by Joseph Welzenbaum. ~~ 


14winner, p. 126. 
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beings in controlling their technical ereetions ute any. computer systems or perier 
technical systems), BuOnmous technology enables us to convince ourssives that 
the system progresses Independently « of our actions, and 1 thereby to deny pene: 
responsibility for our actions. 


it disturbs me to be told that technology "demands" an 1 action 
the speaker: favours, that: "you cannot stip progress.” © it-trow | 
bles me that we are so easily pressured by purveyors of tech |. 
‘nology Into - permitting so-called: “progrese*: to after-odr lve | 
without attempting to contro! it- as if technology were an 
“Irrepressible force: of nature ‘to. which we must meekly aubmit. 
If we reflected, we might discover that not everything hailed as 
progress: contributes to ‘heppinsss;: i the- few isnot always. . 
Detter nor the old always outdated: 


Finally, ina society in which technology has become an autonomous force, 


sila only ruling princes appears to on that the ‘technological system, must be 


Sek ge ~ she rae ie 


expandéd: at whatever cost. Humen agents are permitted to mere decisions only 
according to criteria related to maximizing technological ‘mctency; Dut: this ls not 


real choice. 16 
To maximize energy, speed, or automation, without reference to 
: the - complex . coriditions ‘that-suetait ‘organic life; have beconie 
ends in themselves .... Under the pretext of saving labor, ; 
the ultimate end of this technics is to displace fe; or rather, to ~~ 
transfer the attributes of life to the machine and the mechant 
Cat collective; allowirig nomenpuete —- 
as ; may be oonsroned and Aeisegaeractes 


more uses the phrase "the new “Mogatectnics” to describe belied modern oe 
whines. chief _ purpose is control. over y the » physical world, ania ultimately over man 


himself. 


Eran H. G. Rickover, “A Humanistic Technology," in Technology and 
a peding dae ‘Nout: de: ‘mavens: Ceaas ne — ‘Company, 1972), 


16 eu, p 80. 


17 Lewis .Mumford,.."Authoritarian . and. : Demberntic Technics," In Technology 
and. Culture, ed...by .Melvin . ore and William . H- bavenport eae York: 
Schocken Books, 1972), P. 56.. 
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4.2: The Computer Revolution 

Computers are perhaps the most powerful pieyinee generated by modern 
technology. Unfortunately, computer systems are often used simply because they 
are there. We are too frequently guided by the technological ‘maxim that any new 
technology that is possible must es enene be utlized, whether or not it Is needed 
or desired. . 

{f a machine can yield a given result, it must be uscd to capea- | 

city, and it Is censiderad. criminal.:and..entieagial. not..to-do -so. 

Technical automatism may not be judged or questioned; immed 

ate use must be found. for the mast recent, efficient, and. techn 

ical process. 
’ Thus, Butler Lampson explains that we will ses increasingly large computer pro- 
grams in the future, paceuse of the availability of more and better computer 
Americans will. indeed find a need for. computers: at hom. “because, technological 
advances are being made so rapidiy. 20 ‘one might ak why technological advances 
are being made so rapldty. ant discover that. the eaplaity. of: technological advance- 
ment is encouraged by the a prior! perception. of ‘s-sopletal need for the fruits of 
this advancement; thus, the minute. size of thie; vicioue, ‘elrcie. is exposed. What is 
technically feasible. is allowed to happen: without regard te cansequences. 

We have already seen that. ecenselc. Jncontives-and sociel pressures com- 
bine. to. demand the use of computer systems, This situation has reached a point 
where any answer. generated: by a-computer is often-acgeptable to us because of 

18Enui, p. 80. 

18putier Lampson, "Building Programs," MIT Laboratory for Computer Science 
Distinguished Lecturer. Series, May 1, 1971. 


20 mitchell Lynch, “A Computer Error: Trying to Use One in Your } ome, " Wall 
Street Journal (May 14, 1979), 33. 
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its automatic component. The worst charge of a technological society is that of 


the impedance of technical automatism. in a very real sense, we cannot afford to 


doubt our computers. 

The so-called computer "revolution" consists eraely of computer aspucalions 
that are primarily guided by an implicit poket that ‘computers ‘should be used. This 
assumption is evident in many writings on the use of computers in psychotherapy, 
education, and military situations. Writers appear to start with. the unjustified 
assumption that computers ought to be used to solve a given probiem, and proceed 
to select for further consideration only those aspects of the problem that appear 
computable (refer to the following section for a- discussion of how “the problem" Is 
chosen In the first place). The problem itself almost appears to merit a. status, 
secondary = that of the computer. 


Data models are tools. They do not contain in themselves 
the “true" structure of information...... He {a user] has to 
learn how to use It. We generally presume that this learning is 
required only because of ‘the complexity of the ‘tool. - Difficuities 
are initially perceived as a faluse to fully understand the 
theory; there is an expectation that perseverance will lead to a 
marvelous insight into how the theory fits the problem. in fact, 
much of his “learning” is really a struggle to contive some: way 
of fitting his problem to the tool: changing the way he thinks 
‘about his information, ‘expetmenting ‘with © different ways. of 
representing It, and perhaps even abandoning some parts of his 
intended appfication because the tool: won't handle it. Much. of 
this "learning" process is really a conditioning of his percep- 
tions, so that he fearns to accept..as' fect these assumptions: 
needed to make the theory work, and tg! sce or sine as 
triviel those caees ‘where the theory Tails: 


The question of whether a computer system is: the best solution, or even an 
appropriate solution, to a prebiem, and the question of whether we should direct our 


efforts elsewhere Instead of forging ahead with another computer. system, are not — 


21 winiam Kent, Data and were Ghew York: North-Holland 1 Publishing ‘Com- 
pany, 1978), pp. 194-196. 


— 
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always asked when they should be. 

The usefulness of the computer Is often assumed perhaps because of the 
awe with which our society generally regards science and. technology. in our 
‘eagerness to find solutions to. perpetual problems, we. turn to what we believe we 
do best; in modern society,. this is determined. by the production of solid results. 
",.. in the West at least, the test Is not so much whet. do you know? or how 
elegant is your interpretation of worldly phenomena? but rather what can you actu- 
ally dopu22 Thus, we assume that a theory of any interpersonal activity can be 
expressed in the form of a computer program; that improved means will triumph 
over carelessly considered ends; and that what @ppears. fq the average person as 
the formal eloquence of the computer wad tke euatem behind It. has the power to 
transmute errors into truths. 

It is in the realm of social difficulties that the computer revolution Is particu- 
larly inappropriate. One computer scientist. has described a pot-too-distant future in 
which we will have access to a carota purpose computing language that can 
describe any system that can be imagined.23 In this way, we are encouraged to 
believe that a more fully computerized world will eliminate ‘existing social problems, 
"... to delude [ourselves] that gigantic instruments can take the place of no 
ideas at al.24 What we are not encouraged to do is to come to grips with the 
human sources of these problems, to consider how subjective, Interpersonal factors 
may be computed, or to axaiiiie criteria according to which social problems may be 


considered "solved." 


22winner, p. 26. 
231 ampson, op. cit. 


24 joseph Weizenbaum, "Human Choice in the Interstices of . the 
Megamachine," p. 14. Lecture presented at the IFIPS Conference on “Human 
Choice and Computers,“ Vienna, Austria, in June, 1979. 
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Norbert Wiener points out some of the important ways: in which social activi 
ties differ from the activities of computers: 


. . learning machines must act according.to seme norm of good 
performance. in the case of game-playing machines, where the 
' permissible moves are arbttrarity eetabiiehed :n.-advance, and. 
the object of the game is to win by a series of permissible 
rules according td a strict convention: that: determines winning 
or losing, this norm creates no problem. However, there are 
many activities ‘that we should ike: ‘to “imprave by ieerning 
processes in which the success of the activity is itself to be 
judged by a criterion involving human: dveings,. and: in which the 
problem of reduction of this criterion to formal rules is far 
from- easy.* [! weuld add thet ‘such: reduction is usually not 
possible at all] 


To assign what purports to be precise values to such 

essentially vague quantities is aéither useful nor. honest, : fils 

any pretense of applying precise formulae to these raed 

defined quantities is a sham and a waste<of: time.“:. [my 

emphasis] 
Abbe Mowsnowl2 criticizes “automated common sense" - the substitution of formal 
processes for the intuitive decision making of ¢ an experienced manager. | It Is smely 
not always the case that all - or enough - of this intuitive knowledge can be wade 
explicit; more often than not, valuable information Is lost in the translation to ae 


language of the computer. 


.4.3: Defining the Problem 
. A noticeable effect of the computer revolution ls the frequent transformation | 

of human difficulties Into a form that is amenavis to a computerized remedy. By 

omitting the step of convincing ourselves that a given problem realty is technologi- | 


cally based, we are committing ourselves to forcing some problems. into a 


25norbert Wiener, God and Golem, Inc. (Cambridge, MA: The MIT Press, 
1964), pp. 76-77. ° 


26ipi0., p. 91. 
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mold - that of a computer system - in which they may not fit. The nature of such 
problems can easily ba -aubdiied by the method of approach and the techniques 
employed. 

Many writers have remarked that the technological society Is one in which 
our needs and desires are inevitably formulated as technological cient it Is 
the solution, e.g., the computer, that defines the problem. | 

If the technique in question is not exactiy adapted to a pro 

posed human end, and if an individual.pretends that he is adapt- 

ing the technique to the end, It Is generally quickly ee that 

it is the end which is being. modified,.nat: the. technique.~ 
Writing about a more specific domain, Philip Kraft. comments on the attempt to for 
malize the activity of programming by increasing the use .of pre-tested software 
routines: “In effect, the use of canned programs represents a joint decision by | 
software sellers. and software buyers to make the problems. fit the solutions at. 
hand."28 Note Kraft’s perception of the fact that the exigting situation Is. the 
result of decisions that have.been made by people... ... 

The area in which the.computer shows up most glaringly as a solution looking 
for a problem.is that of social. problems,.in. which thera is most often no perceived 
need for: computation. (for example, the "problems". that characterize our educa- 
tional or legaj: systems). Quick. tachnological: fixes far.social problems have..a way. 
of affecting. and.changing the. problems..in.unforeseen..ways,...Furthenmore, just as. 
the application - of -patches. .to ..computer programs. compounds thelr 

27eitu, p. 141. 


28 pnp: Kraft, Programmers and Managers: :The: Routinization of — 
Programming in the United States (New York: Springer-Verlag, 1977), p. 35. Note 


that. this: difficulty. is - common:: to all ’ situations in «~which:.standardized:- not: - 


personalized - solutions are applied to problems. | believe that_Kraft is primarily 
concerned with canned databases, management Information systems, etc., and not 
with relatively ‘harthiess ‘tand' geriutnely ‘fniportant) : “whips ‘suth” ‘as: “canned 

mathematical routines. . 
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incomprehensibility, so the use of an entire computer syetem as a peten solution te 
a deep human difficulty can set in motion a pac ahd propagation o of incomprehenst - 
bility. 

in considering the application of comeucer? to social RrOpIGms,. we should 
first contemplate our perception of human difficulties as Tprobiema:< sosee? 
Weizenbaum has pointed out that problems such as a series: of mathomatical equa- : 
tions have permanent Pavone; but that this is not the case “with interpersonal 
difficulties. These problems involve -conticts of Interset between people and can- 


not be undeietedd solety in: information: procéiaaing: tering; trese préblaiie “cannot be 


understood ‘without ‘first understanding peopte. Human problems (for instance, sys- ~~ 


tem Incomprehensibiity) are not’ "solved" in: thé computational sense of the word; © 
what would constitute a solution to a social problem? Rather, they are transformed 
into ottier problems that may be easier to live with: than those they replace.2? 
We fail into a mode of problem solving when we realize, at some subcons- 
clous level, that in the realm of soctal difficulties; the subjective nature of problem ~ 
definition renders problems not just complex, but ‘extraordinarily: difficult to deal 
with. Comiputer systems: are comparatively easy to deal with. ‘Thus;'we concen - 
trate our energy on improving: military’ commrarid: and control’ syetemeinetead of - 
questioning the néed' for ever more cdmpléx-end advanced forms’ of destraction; we — 
extol the virtiies of computerized psychotherapy withetit @xiininttg the: reasons ° 
why Increasingly large numbers of peopte seek peychotherapeutic —_ and ‘we ey 
to introduce computers into schools on a massive scale without ever facing the 
reemty: of. what is. happening in mmencen' Schon: ateoned nen sdisigleac yan tteoreth 


cal ae “We-can convinoe ourselves 4 that computers can solve. or problems as 


2capeh, Weizeabaum, “The ‘Last Oream," The ‘Conterence. Board. Magazine, 
XIV, No. 7 (July, 1977), 41. 
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well as or better than people only by substituting mechanical gadgetry for human 
attention and by practicing interpersonal activities in inhumane ways. “The world 
becomes computerized when all human problems are reduced to technological prob- 


lems."2° 


4.4: The Reduction of Human Experience 

Perhaps the most regrettable effect of the technological atructiririg of 
.Present day society is the reduction of people | to Inforination processing organiems: 
Intelligence, once thought to be the exclusive domain of human beings, is now 
often defined operationally, so that we may. vee of intelligent machinea. At the 
1977 International Joint Conferences on Artificial Intetigence, Al researcher Edward 
Fredkin stated that the achievement of a SHnKing machine" requires a scompmanen 
of only enaneong and science.?" Simon and Newell's General Problem Solver con 
puter program is an attenet to maptoment their bellef that me elementary 
‘processes underlying human thinking are analogous to the Information processing of 
a computer. 82 Dartmouth president John nemeny sees no good reason 1 not 
assuming the intelligence of computers, because they manifest intelligence In a 
scientifically testable (which is to say, extremely limited) énéda. Remany equates 
apparent randomness (what we might call incomprehensibility) In a computer system 
with intuition and creativity In people, and implies that random computer behavior is 


as desirable as creative human acts.°° 


30 Joseph Weizenbaum, “Apollo Agonistes," lecture presented at SUNY in 
Albany, New York, April 20, 1979.- — 


Slisraei Shenker, “Man and Machine Match Minds at MALT.) " New York © 
Times, August 27, 1977, p. 8, quoting Fredkin... oat ot 


32Herbert A. Simon, “What Computers Mean for Man and Society," Science, 
195 (March 18, 1977), 1186. 


33 john G. Kemeny, Man and the Computer (New York: Charles Scribner's 
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What | object to in such observations is thelr pect reduction of people 
and of human experience; their identification of “the scientific conception of ‘valid — 
experience with the whole of existence."24 Rdaxak nates, that In ‘equating people 
with machines, we can either alae iactiltes up to our level or ower ourselves; he 
laments that we have done the latter - that we have reduced all human culture to : 


the machine’s limited capabilities. 86 Roszak is hardly sone! in i Percenvna the oe 


Soe. 


dual Shahan of human needs: noeres and thought processes according to the ; 


demands of technique; for Instance, Winner writes of the shaping of puman ‘con 


‘ : sf 
sciousness within narrow technical ehatinals. 36 


In technological terms, an individuar's social worth is proportional to his "pro- 
ductive capacity in a competitive labor market. 37 The Production norms acute 
by the ‘goal of profitability étten cohflict with spontanetty and berecnal creativity. | 
individual participation in the technological ee, ts tolerated only ‘according to 
the degree of an individuat's subordination to ‘the search “for efficiency: only that. 


- which is controllable is allowed to remain in | man. Kemeny suggests that we may 


have to ‘accommodate the computer systems that will play an increasingly maior 


Weis 


I have siready mantioned: that life. in modern society is formulated as a suc- 


Sons, 1972), pp. 11, 18. 


34anbe Mowshowitz, The Conquest of Will: information Processing in Human 
Affairs (USA: Addison-Wesley Publishing Company, 4076), p. 276. 


35poszak, p. 533. 
86winner, p. 127. 


87 mowshowitz, | p. 250. In reference to the emphasis. ‘on the economic. value 
of people, consider the following: "But where human lives are at stake, and particu- 
larly when these people have paid for their. babel eabeubeea a precincts Ae oper 
degree of safety is required." Thompson, p.:'@7. ==: 


38x emeny, op.cit. 


64. Chapter 4: The Technological Society 


cession of.problems to be solved. Humans are viewed as problem solvers only, and 
are encouraged to resist sloppy human reason and intuition In favor of the artificial 
reasoning of machines. Mazilsh urges us to see man’s nature as being continuous 
with his tools and machines.22 People become interchangeable as they grow 
increasingly alienated from their work and from other people. Technological reduc- 
tlonism can have nothing but an erosive effect on the seif Image of people. 

it is not only the human being, but the entire human experience which is 
viewed. from a reductionist viewpoint. Ethical and moral traditions that are not cost 
effective become obsolete; beauty becomes that which is well adapted to use. 
"The virtues-of slow information processing and labor done at a leisurely pace have: 
long since been sacrificed to the norms of work appropriate to the electronic exem- 
plar. The idea that a task is something to be pondered or even savored Is entirely | 
foreign to this mode of activity."40 

In order to accept a perception of life limited by technique, "humanity ... 
has or will soon have transferred all its attention to one aspect of its being - It 
has sacrificed emotion for rational thought."41 Thus, Herbert Simon identifies our 
most challenging problem, “the scientific problem of our age - how to understand 
ourselves more deeply."42 In identifying interpersonal understanding as a scientific 
problem, Simon is severely limiting the role of human beings In the most human 


endeavor of all - understanding ourselves. As Ellul wrote, "men do not need to 


39Bruce Mazlish, “The Fourth Discontinuity," In J. Mack Adams and Douglas 
H. Haden, Social Effects of Computer Use and Misuse (New York: John Wiley & 
Sons, 1976). 


40winner, p. 205. 


41¢. ¢. Gottlieb and A. Borodin, Social Issues in Computing (New York: 
Academic Press, 1973), p. 266. 


42 Shenker, quoting Simon. 
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understand each other in order to carry out the most important endeavors of our 


times."43 


43eiul, p. 132. 
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Chapter 5: Results of incomprehensible Systems 


... technique, as a result.of the perfection of means which it 
has placed at the disposal of modern man, has effectively 
suppressed the respite of. time indiepenaable to the. rhythm of 
life; between desire and the satisfaction of desire there is no 
jonger: the duration which is necessary-.for real oheice and 
examination. There is no longer respite for refiecting or choos- 
ing or adapting oneself, or for acting or..wishing or pulling one- 
self together. The rule of life is: No sooner sald than done. 
Life has become a racecourse compased of instantaneous varia- 
tions of the universe, a succession of objective events which 
drag us along and lead us astray without anywhere affording us 
the possiblity of standing apart, taking stock, ant oeesinG to 
act. 
Sachues Etlul 
- The use of computer systems, many of which are incomprehensible to the: 
people who work with them, is firmly established in today’s society, so It Is not 
surprising that these systems have had significant Impacts on our society of users. 
t have already examined a variety of factors that help explain why, in many cases, 
‘these impacts are undesirable and unanticipated. For example, consider the follow- 
Ing points: first, our motivations for using computer systems are often not related to 
the nature of the problem at hand, and second, our perception of problems Is fre- 
quently distorted by a pre-determined bias in favor of technological solutions. 

In the previous chapter, | discussed some characteristics of a technological 
society; In this chapter, | will narrow that discussion to a consideration of some of 
the effects of the widespread presence of large computer systems. ‘Some of these 
effects have already been expe ce In my dlacussion: of the sources of 
incomprehensible systems; this Is ‘appropriate because technological systems often 


nurture attitudes and create conditions which support. ‘thelr continued use and 


expansion. Thus, certain issues | have previously ‘examined - for instance, 


T Jacques Ellul, The Technological Society (New York: Vintage Books, 1964), 
pp. 329-330. 
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autonomous technology, Lite p tage meees: and the maintenance of what is 
often only an illusion of complexity in Bainniitat syateue: - are factors which both 
lead to the generation and use y of: oe vo. “ari: result from our r- continued 
utilization of ae dependence ‘on: these systems. in a satiety ‘dominated by a 
myriad of Garcia, computers: oceupy « uniquely sweineptring: position. This 
chapter is an examination of some ditneutt Westions raeed ‘by oar growing use of 


aa Tf 


computer systems whieh we do! ‘net Unidorstand "among shams ‘questions about 


dependance on automattesity omerated: tout ane: responotbiity for: decisions 


a 


mace wth the aid of Conputene: 


5.1: Rationality th 
’ : he uae 


Scientific explanations cerive no thelr F ne and thelr imitedness from the 
method of abstraction and simplification by which ectence proceeds. The scientific 
method Is tremendously weetul, but in limited 1 ways; only information which is, in 


Kemeny’s words: "scientifically ‘testable,*? can be utlized in the construction of 


Saya § 


scientific experiments: models, and theories. "Such information constitutes only « one 


ra 
| 


small aspect - the scientifically quantifiable aspect - "of the world. Real life situa 


“ah 


tions are characterized by an extreme richness of “knowledge, much of. which is 


.t 


"unscientific" and hence essentially unsuited for technical manipulation. ue as 


ht 


the reach for technological omnipotence continues to ‘require: ‘the reduction of 
human beings: to that which tearpowgy can | explain and 1 contro 80 ‘the maniteste- 


— of power that we “attribute to. computer technology Jes necessitated the 


q- 
s 


reduction of Scbieiee’| to those with which a computer can deal. 


One key to the seemingly universal applicability of computer systems | as 


oo een id 


2 John Kemeny, Man and the ee snow Yorks Charles Scribner’s Sons, 
1972), p. 11. 
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solutions to problems is that before the application of computers, the problem 
domain is limited by being "rationalized." Problems must be made sufficiently expi- 
cit for interpretation by a computer. system; to computerize often means first to 
rationalize. In many cases, it Is this initial organizational effort, and not any conr 
puter system at all, that serves as the échidon to the problem. Moreover, if the 
large expenditures of time, effort, and money needed to set up a large computer 
system result in a colossal failure, this failure is usually not linked to the question- 
able appropriateness of using computers In the first place; instead, a new problem 
is formed - that of the insufficiency of the logical composition of the problem area 
(this new “probtem," in turn, may be deemed suitable for computerization). For 
instance, in writing of existing failures in the use of ‘computers in conjunction with 
medical practice, Abbe Mowshowitz states that "the promise is enormous, but much 
depends on rationalizing the organization of health-care services."® The successful 
computerization of health care services depends more. directly on the human effort 
of organizing the field than on the: seoondary step of bringing In computers. 

The dangers of excessive rationalization derive from the lack of considera- 
tion for values which do not seem quantifiable, and the © consequent loss of informa- 
tion in a purely rational planning or decision n making process. The limiting rationality 
that computer systems demand encourages us to disregard the most difficult - that 
is, non-computable - aspects of a problem.: Welzenbaum. emphasizes the fact that 

‘< 


computers process only information, not meanings. Technique requires that certain 


aspects of problems be ignored; and even more significantly, determines which 


Svowshéwitz, The Conquest of Will: Information Processing in Human Affairs 
(USA: Addison-Wesley Publishing Company, 1976); p: 123. x 


“Weizenbaum, “Once More: The Computer Revolution,” In The Computer Age: 
A Twerty-Year: View, ed. by Michaef't. Dertouzes’ ‘and Joel: Moses" Scene MA: 
The MIT Press, in press). 
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problems are to be seriously considered. " ... we have permitted technological 
metaphors to so thoroughly invade our thought processes that we have finally abdi- 


cated to technology the very duty to formulate questions."© 


5.2: Believing Computers | 
Our inability to comorehend many’ existing computer systems n means thet we 
must rely on the correctness of the systems; we have allowed such systems to 
become indispensable to us. 
We can’t count on making such complex computations manually 
when we know that an enemy will ‘use a-computer. So we must 
rely on a computer for speed, which makes our decisions totally 
dependant upon both the avaitability and the sae -of the 
computer when rapid decisions neve to be made. 
The obvious risk is that of the increased mmpect< of oer: errors as we increase 
our dependance on computer systems. . | . 
Today, the population in general does not understand technological forces, 
but is kept submissive and content with ne wide range of services offered. 
Different groups, such as business managers: must t depend on the abate of compu- 
terized information systems, although they do not have the time to supervise the 
collection of data or to satisfy themselves as to the reliability of their computer 
systems (nor are they encouraged to do 80). . | 
. how utterly dependant we have become on our electronic 


super-tools, fiow essential we ii 
not that they were needed in the Sat vane. 


SWeizenbaum, “On the Impact of the Computer.on Society," selene 176 
(May 12, 1972), p. 622. 


- Sioward Thompson, Joint. Man/Machine. Decisions (Cleveland, Ohlo: sye- 
tems Procedures Assoc., 1965), p:. 40.- 


7Weizenbeum, "Human Choice in the interstices of the. tiecemaching.* Lec- 
ture presented at the IFIPS. Conference on “Human Cheice and Computers," Vienna, 
Austria, in June, 1979. ; = 
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The issue of dependance takes on a special meaning in ‘relation to 
incomprehensibility. We have already seen that large computer systems are too 
complicated for anyone to directly monitor their operation. Thus, our day-to-day 
contact with computers must proceed largely on the basis of our belief in their 
correctness. When the computer systems we use are not understandable to us, 
this belief reduces to fatth in a technology that we have been taught is too compli- 
cated for anyone but a specialist to understand. We abdicate the responsibility for 
our decision making to a technology that frequently is not comprehensible, while at 
the same time attempting to maintain a feeling of control. Thus, we assure our- 
selves that "these versatile machines have become the galley slaves of capital- 
ism."® .The real situation, however, is that our sense of control is largely illusory, | 
and that we have been and continue to be largely unjustified in transferring respon- 
sibility to computer systems. This was made abundantly clear during the Vietnam 
era, when the Chairman of tke Joint Chiefs of Staff regretted that "It is unfortunate 
that we have become slaves to these damned computers."® 

Many people are now making decisions to some extent on the basis of 
potentially unreliable computer generated output, and some of these decisions can 
have vital, nonreversible impacts. In some cases, "... computers can provide not 
only the information on which decisions are made but can themselves make deci- 
sions." 19 At the very least, we must ask what kinds of decisions, if any, computers 
ought to make; we must decide whether the increased risk of error is worth the 


alleged gain in precision and rationality, if we depend more and more heavily on 


SuThe Computer Society," Time, Vol. 11, No. & (February 20, 1978), p. 50. 
- 8ip14., p. 45, quoting Admiral Thomas Moorer. Tobe ash 


10herbert Simon, What Computers Mean for Man and Society," Sc/ence, 195. 
(March 18, 1977), p. 1187. 
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computers; and we must worry that the goals of a computer system on which we 
rety may not be our own goals: 

A goakseeking mechanism will not necessarily seek our goals 

unless we design it for that purpose, afd in that designing we 

must foresee all steps of the process for which it is designed, 

instead of exercising a tentative foresight: which goes up to a | 

certain point, and can be continued from that on as new 

difficutties arise. Tha’ périaltiés for érrors Of foresight, gréat as 

they are now, will be. eno increased as automizetion 

comes into its full use. . 
Langdon Winner warns of “the distinct possiblity of going adrift in a vast’sea of 


unintended consequences. wt2 


5.3: Technique and Morality 1% 

{It is to a large extent the common perception of computers, which may not 
have much relation to actual computer systems, that has determined the ‘degree of. 
our rellance on computer systems ‘and the extent to whith we have: transferred: 
human responsibility to automatic systeins. The issue of responsibility can refer to 
different things in relation to computer systems; for instance, the responsiblity of | 
people for the propagation of new systems. i Considering the concept! of auto 
nomous technology, we saw that often new systems ate constructed without expi- 
- elt or conscious human approval. Not only do we have trouble comprehending the 
systems we use, but it is often dimcutt for u¥'to inderatand how and why they are. 


created. 


11 Norbert Wiener, God } and Golem, inc. (Cambridge, MA: The MIT Press, 
1977), p. 63 


— Winner, Autonomous Technology (Cambridge, MA: The MIT Press, 
1977), p. 89. 

18«Technique never cbaerved ‘the distinction between moral and immoral 
use. It tends, on the contrary, to create a completely: independent: technical moral 
ty." Ellul, p. 97. 
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Moira is at work here - a fate that employs the free action of 
men to bring about ends that carry an aroma of necessity. 


The modern emphasis on a scientific analysis of behavior undermines personal 
autonomy by placing control of human actions in the environment. Responsibility for 
the unbounded increase in our use of computer systems seams to fail only on a 
technology that is, or so it is claimed, value free. . 

_ Technological elitism is at. the root of much of the widespread avoidance of 
responsibility for computer systems. We are taught that only the experts can 
know what is best for us; in Kenneth Laudon’s words, there has been “a denigra- 
tion of faith in the wisdom of ordinary citizens." 15 Increased _centralization, which . 
frequently results from the introduction of a computer. system, formalizes and 
rigidifies the prominence of those people who can claim. to. understand computers, . 
In Social Issues in Computing, Gottlieb and Borodin comment _.an the political power — 
of technocrats: because politicians themselves, uayally have no technical expertise, 
it is the technologists who define the alternatives. for all of us.'6 

. The people who appear to be most directly accountable for complex com- 
puter systems - computer scientists and the researchers who determine. the state 
of the art - do not always manifest attitudes that are as cautious or as humble as 
their positions seem to dictate. Joshua Lederberg, for example, has said.that there 
is no difference between the things computer should not do and the things people 


should not do; the only important thing is to be sure the machines do not get out of 


14winner, p. 71. 


18xenneth C. Laudon, review of The Conquest. of Will: laformation. Process- 
-ing in Human Affairs, by Abbe Mowshowitz, in Science, 163. (September, 1976), 
p. 1111. 


18¢. ¢. Gottlieb and A. Borodin, Social issues in Computing (new York: 
Academic Press, 1973), p. 223. 
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control. ! ? It is not clear whose control Lederberg believes computers are presently 
subject to; many other people belleve that computers have already grown beyond 
the limits of our control. Still others believe that some programs themseives are 
already exercising contro!: Herbert Simon says of automatic process control sys- 
tems that “their programs retain contro! over the ongoing process."15 When ques- 
tioned about how far artificial intelligence systems could go, Simon, with apparent 
disregard for issues of responsibility, incomprehensibitity, reliability, appropriateness 
of use, etc., replied only that "we'll know that when we're done.*19 
Since, as we have seen, a large system has no identifiable group of authors, 

there is usually no one who feels directly responsible for the output of the systems 
and for decisions which make use of that output. Currently, accountabifity for the 
reliability of computer systems is so vaguely defined and so well “distributed” that 
It Is fundamentally nonexistent. This is so despite the fact that many people are 
concerned about our tendency to allow computer systems to become ultimate 
authorities which require littie justification. 

If the activities carried out by computers cannot be readily 

monitored and guided by peopte, and if human processing of 

information cannot be easily intermixed with computer process- 

ing, coppiters: tend to become unchaliengeable author- 

ites . 

Recall the discussion of theory of behavior in Chapter 2, where | noted that 


comprehensible systems are likely to be founded on weil understood theoretical 


1?iee Dembart, “Experts Argue Whether Computers Could Reason, and if. 
They Should,“ New York Times, May 8, 1977, p. 34. 


18 terbert Simon, "What Computers Mean for Man and Society," Science, 
195 (March 18, 1977), p. 1187. 


19)p1¢. 


20Robert Fano,.“On the Social Role of Computer Communications," Proceed- 
ings of the IEEE, 60, No. 11 (November, 1972), p. 1261. 
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bases, so that they serve as models of a theory and not as unquestionable authori 
ties. In the case of incomprehensible systems, we have noted that there Is fre- 
quently no well based theory of use; the system itself is the theory of its use. 


' Thus, whoever doubts the system finds himself in conflict not with a theory but with 


an enormous, incomprehensible programming patchwork. Nevertheless, some people - 


have gone to the extreme of advocating that computers be held responsible for 
themselves; Howard Thompson believes in letting the machine be responsible for its 
share of the decision making load in joint man/machine decisions.*' The question | 
must. ask is what meaning could machine responsibility possibly have in a human 
world? 


The habit of speech, and it surely reflects a habit of thought, 
that makes instruments responsible for events, leads directly to 
speaking and thinking of science and technology as autonomous 
forces and to the Idea of technological inevitability. It leads 
finally to the proposition that man is, after all, impotent to strug- 
gle with powerful impersonal agencies of his own making over 
-which he has fost control, and that he is therefore jus d in 
abdicating responsibility for the consequences of his acts. 


21 Thompson, pp. 27-28. 


22 Joseph Weizenbaum, "Controversies and Responsibilities," Datamation (No- 
vember 15, 1979), p. 173. 
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Chapter 6: What to Do About incomprehensible Systems 


We must return to the human center. We must challenge this 
authoritarian system that has given to an underdimensioned 


ideology and technology the authority that belongs to the human 
personality. | repeat: life cannot be delegated. ; 
oe Lewis Mumford 
“| would like to end my discussion of incomprehensible computer systems on a 
relatively positive note; in this final chapter, | consider suggested means of dealing 
‘with the existence of incomprehensible systems... The chapter begins with a dis- 
cussion of program verification techniques that is more technical than that encoun- 
tered in the rest of this thesis. Even in this area of formal study, many of the 
difficult problems are not technical ones and cannot be solved solely by studying 
computer programs: for example, the problem of hour to specify what a program is 
supposed to do. This chapter, like previous ones, expands from a program-oriented 
viewpoint (which in this instance verification studies exemplify) to one. concerned 
primarily with systems (both technical and social). 
It should be noted that the following discussion pertains even to comprehen- 


sible computer systems; in fact, to all.modern technological systems. Examination 


of well understood systems will. generally reveal that many of the programming. 


techniques described below were used, perhaps in. modified forms, In the construc- 


tion of the systems. However, evidence of the ethical "techniques" which are dis- 


cussed In the latter part of this chapter (criticism of technology, acceptance of | 


individual responsibility, and humanization of technological systems) is harder to 
find, even in systems that we might not label technically incomprehensible. Deal 


ings with systems that are incomprehensible do ralse unique ethical problems, 


lLewis Mumford, “Authoritarian and Democratic Technics," In Technology and 
Culture, ed. by Melvin Kranzberg and William H. Davenport (New York: Schocken 
Books, 1972), p. 58 
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because It Is In these Interactions that the necessity of trusting computers Is Implt 
cit, and our vadnarenaity Is aeeu ieee | _ | 7 

We already know that incomprehensibility in ‘computer ayetome can be mani 
fested in many ways. A social form of. technological incomprehensibitty is reflected 
by the pervasive sense that most people have of not occupying a eeaninatil posi 
tion In modern technological society. For the most part, we do not underetand how 
our technological systems operate - we do not know’ how to onitieize technology, 
how to judge the extent to which we should: depend on It, etc. It Is especially 
because of these social effects of Incompretensibility that we feel no contro! over 
the use and expansion of computer. technology and are unwilling to assume respon 
sibility for It; it Is these issues that are ‘addressed in the second half of this 


chapter. 


6.1: Program Verification 

Verification has proved to be a difficult term to satisfactorily define, largely’ - 
because of the human ‘féctors Invoived, first In the creation of the programs which 
are to be verified (programs carry with thém. their programiérs’ intentions, which 
are often unclear), and then in the interpretation of the ‘proof of correctness (we: 
want to be able to ‘trust’ programs, but we each have different criteria for bellev- 
Ing a correctness proof). One suggested definition is the following: program 
verification Is that branch of computer scleneé: whose goal is to establish “whether 
a [given] program performs its intended task."* What remain unanswered are the 


questions of what does It mean to talk about the intefitions of a programming task, 


2Barbara H. Liskov and Valdis Berzins, “An Appraisal of Program 
Specifications,“ MIT Laboratory for Computer Science cse ‘Memo 147 (July, 1976), 
p. 2. 
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and what does the assurance offered by a verification proof buy us? 

The need for a discipline like program verification springs from two conditions 
that have existed for some time: (1) professionai dislike for flaws In finished pro- 
ducts (in this case, computer programs), and (2) increased dependance on pro 

- grams that can have extremely destructive effects. On the one hand, there is a 
desire for certainty that has led us to mathematical formalism for-a certification of 
program correctness. On the other hand (and more Importantly), now that we have 
created computer systems that affect vital areas of our lives, we are beginning to 
-wonder how we can depend: upon the information that we get from computers. 


Because “even minor errors [let alone grossly misconceived "designs"] can have 


- serlous consequences and be costly to fix,"* the role of verification in increasing . 


our trust in computer systems Is both methodologically and ethically Important. In 
terms of both physical and social costs, we cannot afford to trust unreliable com- 
puter software. 

It is important that the reader recognize the Inherent limitations of the 
verification approach to computer systems, before launching into the following dis- 
cussion. Verification studies are directed toward ‘computer programs. Verification 
researchers are committed to eliminating the relatively low level, technical form of 
incomprehensibility ‘that is characterized by program errors. Recall from the first 
pages of this thesis that it is the incomprehensibility of systems with which | am 
primarily concerned; . this is a much more subtle difficulty. than that addressed by 
work in the field of program verification (in fact, as | have tried. to explain, it is not 


a technical issue at all). Although there is a significant role that verification proofs 


Ssusan L. Gerhart and Lawrence Yelowitz, “Observations of Fallibility in Ap- 
plications of Modern Programming Methodologies,” /EEE Transactions on Software En- 
gineering, Vol. SE-2, No. 3 (September, 1976), p. 195. 
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can fulfill, exvepcenet from a proof of correctness to a meaning? statement 
about the comprehensisiity of a system (and hence about the appropriateness of 


trusting that system) is, at best, unjustified. 


6.1.1: Automatic Verification 

The most widespread basis for verifying programs sis ve inethod of intermedr 
ate assertions. Tags are > placed at “ey points throughout a program (0.94 loops), 
indicating the state that the program is supposed to be n whenever it passes 
each point at execution time. ». The central idea is that ali loops in a program must 
be "broken"; je., it must not be possible to do a loop Iteration without going 
through a tag. The. Intermediate assertions may relate values of program variables 
at intermediate points to initial _ values or Sa ultinate values. ne Program 


May 


specifications for the initial, state form the frst assertion, and the donee output 


conditions form the final assertion. A proof of program Rei oe divided into a 
number of smaller proofs met a program coming com ‘assertion a wil always satisfy 
the conditions of assertion n+7. The combination of these intermediate proofs 
establishes the partial cotractness of the entre Program. = ee 
Automatic: methods of Program wermaa en: center. oe. the mochamzaton 

of the ooo approach (which might Se called din intemal. hacer sie _approach) 
through the use of sutomauc theorem proving programs. — 

The inductive assertion method reduces program correctness to 

‘a fhhité set of finite paths. “A progtani' path starte With an’infdal 

assertion, continues with executable code, and terminates with 


‘@ final asstrtion. For éactr progr ee. there“fs & topical for 
mula called a verification cond 


The first quest is for a system that would automatically genarate | the verification 


“Ben Wegbreit, "Constructive Methods'‘in® ‘Program - Verification," ‘Xerox Palo 
Alto Research Center (December, 1976), p. 6. 
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conditions. Once that Is done, the proof of the correctness of a program reduces 
to a proof of a theorem in the first-order predicate calculus (higher-order systems 
are also. being examined, but less effort is being devoted to them). The theorem 
establishes the partial correctness of a program, so a separate termination proof is 
- required (in most discussions of program verification, it ls deemed easier to divide a 
total correctness proof Into two proofs, one of partial conectess ~ or correctness 
assuming termination - and ‘one of termination). “According “to this view, a 
verification system consists of a verification condition generator and a theorem 
prover. In practice, the step of proving that the verification conditions are true 
has constituted the bulk of work in this area. The preliminary step of finding 
appropriate intermediate assertions Is still too little understood ‘to be automated. 

There are a variety of problems, both technical ‘and. social, facing jeseureh: 
ers in the field of automatic program verification. Some of the technical difficulties 
may only require further study to be overcome, but some of the constraints 
imposed by less formal problems may represent inherent ‘obstacles to the success 
of verification proofs in Increasing our trust in computer ‘programs. Significant 
difficulties that are currently being addressed include the following: - 

@ It is the hope of verification ieanarchers that the ‘verification of a compu- 
tation is much easier than the original computation. However, ‘time constraints place 
a heey burden on any verification system. aithough a given system may be able 
to verify a large class of programs, nwe are more interested In what the theorem 
prover can do in ‘reasonable’ time."© Thus, goat a complete verification system 
would not necessarily estety the practice time limits that it would have to be sub- 


ject to. Contemporary systems tend. to flounder on oo programs: which are 


SBernard Elspas, et. a/., "An Assessment of. Techniques for Proving Programs 
Correct," Computing Surveys, Vol. 4, No. 2 (June, 1972), p. 127. 
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founded on deep Meocoms: because the enercn space Is Nicreased mu a pom 
where a proof cannot be generated i in a tolerable length of une (and, even when it 
is generated, it is proevely long and complicated). “We ééacis do not - vat see 
how to prove programs are correct In any reasonably sie manner at the present 
time." 46 | | ss | . ; _ . 3 . 

@ Until now, verification proofs have dealt with relatively simple cases (righty 
restricted Erograimaing languages and ideal machines), and are y Rot yet uP to 
level of complicated programs, where they would be useful. Some relevant techni 
cal issues that are acknowledged as problems but have not been ‘satisfactorily 
solved yet are indeterminacy, parallelism, exception errors (e.g., ‘overfiow “and: 
underflow), and side atfects of a given Proganmmng language. 

In addition, there Is the difficulty of dit rsted errors pend of spaciiving “the: 
behavior of a program when an error a detected during execution uf neseerchers 
In program rellability: attempt to anticipate a range of possible execution errors 
ahead of execution time, 30 that errorhandiing measures can be iemeee In the ~s 


eat YEN BEd 
ginal design of the program. _According to the philosophy undertying reliablity work, 
errors are not necessarily olimineted, but they « are aneoreed 8 and dealt with in 
understandable, acceptable, rerebie aye. . - 

e Some ‘computer scientists are concerned ou the 2 bollevability « of f proots, 
a ieaticl that. would not be well savtened oy a , system that peceles a program 
whose correctness It is to establish as iors and outputs a huge proot that Is even 


less comprehensible than was the original, progen:: Rs verification system's 


S james Joyce, “Human Factors in Software Engineering," in The First West 
- Coast. Computer: Feire::. Conference. Proosedings,. ed.-by- Joho..C. Warren, Jr. (Palo 
Alto, CA: Computer Faire, 1977), p. 61. 


‘Liskov and Berzins, p. 18. 
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response of "QED" is not very meaningful if programmers have no reason to trust 
the system. One primary function of a proof of correctness Is “to dramatically 
Increase one’s confidence in the correct functioning of a particular piece of 
software,"® but before this can be accomplished, confidence in the proof process 
itself must be established. . At. worst, a complicated verification system could con 
stitute an additional layer of. incomprehensibility between a computer program and 
its users. | | 7* 

DeMillo, et. a/., convincingly argue that mathematical proofs come to be 
believed because of the: existence of a social network in which proofé are widely 
read, refereed, published, reviewed, discussed, and finally, : internalized, Para- 
phrased, and used.” Although the motivations behind mathematical proofs and pro- 
gram verification proofs are different, the concept of believability is related similarly 
to both. Even a “correct™ proof will not be used if it Is not believed (belief here Is 
defined in the social sense that DeMillo, ef. al., discuss), and DeMillo, et. al. do not 
believe that a social procaee in computer science ‘analogous to that in mathematics 
Is yet well developed. They also ronind us that “the decision to consider a ‘proof’ 
in detail is often iatusenoed by acne slightly ‘rational. concern < ‘how does the 
problem feerz 10 and urge us to strive for the. ‘simplicity ‘that characterizes the 
most important mathematical iaoraien: and proofs. " Finally, "since any proof, no 
matter how ‘formal, can be interpreted in “different. ways ‘by different ‘people, we 


should be aware that, ‘In a practical: “aense: a ‘derivation’ ‘of a theorem or a 


SRichard A. DeMillo, Richard S. Lipton, and Alan J. Periis,: “Social .Processes 
and Proofs. of Theorems and Programs," Yele eer of Fame: Science 
Research Report. #82 (1976), p. 1. 7 : 


8ipid., pp. 6-9. 
190i 1g., p. 10. 
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verification of a proof [can have] only probabilistic validity"! 1 the classical view 
of Judging things in a strictly dualistic Fashion, As not applicable to verification 
proofs of computer programs. We must be wise enough bs recognize the margin of 
error in any assessment of program correctness, and to require “another view of 
‘reliable design’ ... that more Tully: exploits the social mechaniems* "2 to comes 
ment the view taken by verification studies. . 

@ Computer systems are dynamic entities that may be Gauenly specified. 
The foundations of _ modern “software: eyotene: are mae programs with 
specifications and related scocumentavon Suck ietge than ner code. More inal 
tantly, when specifying a system it is Oren Impossible to siete precisely what is to 
_be done. Typically | some claims are made about eet must gnepeen and others 
describe desirable but less crucial penevicr: a9 in practice, specifications change 
and grow as a programming project progresses, reflecting . restructuring of the ort 
ginal purpose of the system and the original perception of, the problem domain. os 
evolution ofa system to meet me new criteria - sseteby not =“ controtied. ‘The 


incompleteness" and imprecision of the specifications for systems makes _Tigorous 


ey 


verification difficult and the impermanence -. on the ‘specifications ‘reduces the 
rewards of producing such a _Verifioation. “14 


@ The interface between an automatic verification aystem and | a Programmer 


must remain informal. What cannot be completely formalized in this Interface Is the 


purpose of the program, an informal, often unstated criterion. ".. it becomes 
11 Gerhart and Yelowitz, p. 206. 
12ipia., p. 197. 


13charies Rich, Howard E. Shrobe, and Richard’ C. Waters, "Computer Aided 
Evolutionary Design for Software . oo M.LT. Artificial avers Laborato: 
ry A. |. Memo 506 (January, 1979), p. 


14 pig. 
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possible to formally prove consistency of programs with ... formal specifications. 
However, the complimentary step of verifying that a program specification imple- 
ments the underlying concept must necessarily remain informal." 18 We are faced 
with the problem of whether or not our specifications are strong enough to express 
our intentions. The most responsible attitude we can take is that "we can never be 
sure ‘that the specifications are correct." 16 With sich an attitude, we cannot be 
certain that a program that has been verified to be correct (assuming we achieve 
such results some day) will do what we want it to do, unless we are sure that our 
intentions have been accurately and completely codified in the program 
epeciicenons: - 


In verifying a@ program, the system assures us that the program 
satisfies the specifications we have provided. !t cannot deter- 
mine, however, whether those specifications accurately reflect 
the intentions of the programmer. The intentions, after all, exist 
only in the mind of the programmer, and are inaccessible to a 
program verification system. If he has made. an error in 
expressing them, the system has no way of detecting the 
discrepancy. 


Some of the most promising current research. in verification deals with ways in 


which a system could detect the kind of discrepancy mentioned above. '& However, 


at least for the time being and particularly In the case of programs with vital 


consequences, an awareness of this ‘discrepancy’ should play a crucial role in. our 


167Ghar Manna and Richard Waldinger, “An Appraisal of Program 
Specifications,” Stanford Al Lab memo AIM-298 (August, 1977), p. 24. 


18 pig. 
17 Gerhart and Yelowitz, p. 205. 


185, example, the notion of a “programmer's apprentice," which {s “a com- 
puter alded design tool which can help a programmer deal with program evolution 
from the initial design phase right through the continuing maintenance phase." 
Here, the effort is to provide “support during the process of developing code good 
enough to warrant the effort of certification." See Rich, Shrobe, and Waters, "Con- 
puter Aided Evolutionary Design for Software Engineering" (quotations are from 


pp. 1-2). 
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decision to depend on a particular computer program. 


6.1.2: informal Marification 

What Is urged Swrougheut the BU achiah of Sceen verification: is me ee 
that "formalism should supplement, definitely not replace, Common sense and Pro 
gramming Sap ectence: «19 Traditional veritying methods, — may = reduce - 
good ricdcammlia standerces: sou be retained, aoe with. a healthy skepticism 
towards formal preet? of oprrestne®: Intuttion or practical judgment can Getect 


many PLOT A aND errors. one: oF the moet -elomentary insights thet comes yee a 


2% rie; 
Wie? ee Se we 


study of verification is that srecramilne is a human activity and that t there | are 
parts of the programming process that are best handied with some measure of 


informality. © 


fuk 


In the realm oF Informa, martial rotate vertekin, the oot “fundamental 


consideration is a Statement of “whet shoul be Proved’ tease to-yuarsintes that. 


program Is correct... 20 Since program specifications are e generaty the first corr 
tact one has with the ideas that will’ eventually be embodied In the program, Liskov 
, i: °“Phett arguinent ls 
that the increased ‘rigor Of a formal specication ‘facilitates agreement among pro- 
grammers on the meaning: of ‘the’ speciiontion: and, ultimately, the” ‘eatery "OF the 
program. The likelihood of such agreement is Mnecaneed, by the. formal nature of the 


oy eee 


specifications, which should help’ ‘iimit ‘the ‘number and. “scope ‘of possible Interprete- 


and Berzins have emphasized the Value ot forme! spe 


More . specifically, the hase acsutt syntax,,and- semantics of: @ formal 


18¢ethart and Yelowttz, p. 206. 0° 
20 pid. 
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specification language should facilitate the construction of partial correctness 
- proofs for the individual modules into which a program can be composed. This . 
- proof, combined with requirements for module termination, would constitute a 
verification of the program. Liskov and Berzins conclude thelr paper by expressing. 
the current need for proof techniques for the various specification methods. 

There are undeniable benefits from the use of formal Program specifications; 
for example, the abllity to dacompose proofs of program properties, and the ten- 
dency of a formal statement to bring out detalls that, in an Informal specification, 
might easily remain incompletely thought out and ‘hidden under vague descriptions. 
it is important to note that increased formality In program specifications can 
decrease, but not eliminate, the inherent ‘ainbiguity of the early stage of develop- 
ment of a programming project. The informal nature of most existing specifications 
does succeed in presenting the main points in a fashion that Is moré understand- 
able to most programmers than formal mathematical statements.2! Liskov and 
Berzins recognize the role of informal specifications as a Valuable and necessary 
complement to formal specifications. Their ideas can be taken as suggestions for 
Improvement in the practice of software engineering. ‘It is clear that a more 
responsible attitude towards program specifications should be nurtured, so as to 
eliminate a variety of bad programming habits (such as “the common habit of writing 
the specifications after writing the program"@2) and’ the potentially dangerous 
effects they can lead to. imprecise, ‘joosely conceived | design criteria ‘are not ‘tikely 


to support comprehensible systems. 


21 moreover - and this point cannot be overemphasized - most of- the pro- 
grams that | have been discussing all along are in a domain Where formal 
specificationsare -impessibie,. in: considering the. universe -of.. all. eibedhtaatiie applt 
cations, one must question ‘how many of! them-are formalizable, «5: ; 


221 iskov and Berzins, p. 3. 
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The method of intermediate assertions, already discussed, is useful in break- 
ing up the verification process Into manageable units, and in alerting the program: 
mer to particular areas of dace (for example, the error must be In the code which 
lies between assertions 5 and 6). In addition, if the assertions are defined before 
coding, increased modularity is Imposed; the goal Is to divide a large system into a 
number of smailer, more manageable, and. hopefully , more easily comprehensibie 
units. Elspas, et. al, suggest “creating the assertions prior to formulating the 
details of the program algorithm, proving that the assertions correctly express the 
intent of the program, and, finally, writing the code, that lies between the aseer- 
ae it ars ides WISE SE Ry, : 

The difficulty of precisely detning, the Intent of. a program notwithstanding, 24 
the Intermediate assertion approach seems to hold grest promise for successful 
program verification. Jt has already affected the way we write software in @ post 
tive manner, by providing another method of evaluating the meaning of @ program or 
program segment. “There Is no doubt that many. programs could benefit from an 
attempt to carry out an informal. proof, at least for the fact. met such ‘ proof would 
reveal gross misunderstandings In the intended. algorithm.=2 

An extension of the. method of intermediate assertions is .Wegbreit's scheme 
of program justifications. A justification is both .9 program gasertion and a state- 
ment of how the path it applies to: is. to be proven correct. Sustifications help clar- 
ify correctness proofs, but their real value Is that they serve as additional text of 


the program itself, along with code and correctness specifications. In effect, the 


| 23eispas, et. al. p. 142. 


24). do not: mean to dispense: ighthy: att: aiMouttios: tke $hie-one; a0:|-believe 
that they ultimately determine’ the extreme: tmitedness of prdofe of correctness. : 


25eispas, et. al., p. 119. 
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‘ programmer who utillzes this method proves the program correct while coding It. 

Wegbreit’s common sense ideas are instructive both in the study of 
verification and in the fulfillment of present programming tasks. In particular, he. 
- suggests "shifting part of the activity of program verification to /anguage design 

.. and to programming practice"2® and contends that “program correctness Is 
best achieved by explicitly considering the proof as part of the ‘programming pro- 
cess."2" Wegbreit calls for a change ‘In the way programmers think about their 
work — an Increased awareness that reliability is the responsibility of the progran, . 
mer. 

Other researchers have cautioned us not to abandon more “mundane“ 
methods of program verification in favor of newly developed strategies that carry — 
- with them the legitimacy of mathematical formalism. The ‘complex methods that 
researchers may find interesting are not ahenye practical for programmers’ uses. 

The kinds of algorithms that: get ‘ ‘proved’ correct have, nothing 


to.do with software; given a choice between.a. very. good algo 
rithm with a proof of correctness, but which may be hard to 
understand, and a straightforward, unproved:aigoritha which.an 
implementer believes he understands, the complex algorithm 
-Wnvariably loses.. And it is the complex ; + 
interesting and have oe * most chance of bein @ subjected to the 


4. Seclology of ‘proof."2 


In a similar spirit, Gerhart and Yelowitz remark that a common. feature of. program 


errors seems to be a “tendency to concentrate more effort on the harder parts 
which require sophisticated techniques and less _effert.on the ‘obvious’. and easier 


parts,"29 Sophistication should not overshadow..thoroughness. 


26wegbreit, p. 31. 

27 ibid. 

28pemillo, Lipton, and Perlis, p. 13. 
29Gerhart and Yelowitz, p. 205. 
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A variety of ff good programming practices hey evolved that in effect constr 
tute informal enorte to verify sortwars: For mpatance: through the years, mere has 
been an increasing earnest on debugging; now,. wei seveunee debugoing tools 
are commonplace in most large Programming, projects. It is crucial to note oe 
verification methods that have evolved from the practical experiences of program 
mers involve, frst and foremost, an epee Al Progee errors. 

| it Is clear that some common programming nabits: must be overcome. it pro- 
gram. verification techniques are ) to take hold, and that considerable effort will be 
required to do this siiscesatilly, “The. inattvodologles neoeesd to increase 
software reliability are still in their early Stee? of cay ecne ett The toes are not 
easity taught or learned ... 230 For instance, i is avis tose often: he: case ‘nat % 
specifications and documentation: are not regarded as integral parts of a Prog em: 
ming project; yet, the amore that gees] Into eee attemets to clearly state what a 
program is. supposed to so ls essential to the success: ie, any’ verification process. 
Without this ‘thoughtful Planning and documentation cil Programming, reflable large 
scale systems are hot: “possible. ‘Untit the: = mystique’ We removed tro’ -ahiny people’s 
concepts of. ‘Programming ‘and waite Programmers ‘Soine te recognize! the einteal (and 
not just the economic) value of deciendebie software, program ‘vefification will not 


be able to achieve its full potential. 


6.1.81 Conclusions About Verification 
Some common sense ‘corictusions can be drawn-Trom tlie preceding: discussion 


of program verification. The first is really a lesson in humility: “We- San wen 


correctness in our programs, but we must settle for reliability."°7 j Abstract, ideal 


8391p, 
31 pid., p. 206. 
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notions of correctness in terms of perfection are not likely to be fulfilled by real 
computer programs; this must be acknowledged in the course of making decisions 
about the use of “verified" programs. "We simply must learn to live with falllbil- 
ity," and we must learn to do so responsibly, by not treating verified programs as 
perfect programs. “... in mature engineering disciplines, ‘ref//able’ never means 
‘perfect’ ... engineers set probable limits of faliure, relying on other design criteria 
to place these limits well above the conditions likely to be encountered in prac 
tice."33 Current verification methods are best exercised with caution; that ts, with 
an understanding that although they are helpful debugging tools, they do not 
guarantee correctness. 

The second conclusion ts that given a choice between Informal and automatic 
methods of verification, moderation is most appropriate. “Experience with both ... 
should convince us that neither type of evidence Is sufficient and that both types 
are necessary."°4 Verification studies have much to offer in increasing our 
confidence. In computer programs, but confidence Is built up on many levels, and 
different kinds of verification evidence are needed. We come to believe things for 
many reasons; formally structured proofs, informal, intuitive explanations, trial and 
error, and insight all play important roles in inspiring trust In software. The best 
Verification "package" will appeal to as many different channels of knowledge as 
possible. 

Thirdly, computer scientists should begin to place more emphasis on the ethi- 


cal Issues that the use of any vital computer system brings into play. Software 


reliability Is typically measured by the number of "bugs" encountered in a period of 


32peMilio, Lipton, and Perlis, pp. 16-17. 
33 joyce, p. 61. 
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time ("mean time between failures" or “mean time to recover"). We must decide In 
what situations these standards. of reliebiity, and the Systems that cannot be 
trusted beyond these limits, are scceptaple: At Eeee we may want to question 
the basic criteria for the use of Computers In a given application; the degree of 
vitality of the oir of a ayeee. the faveromety of those mph te and the 
comprehensibility of. the system are. some Issues that should be considered. At the 
by. careful .consideration of the ‘famifications of. & dependant relationship between 


man and machine. 


Lastly, | would go back to my first words about verification, at the | beginning 
of. this section, and, state again that: the Pes. of. the Seid of program | verification 
fundamentally miss. the point. In a aor chapter, | I Gecussed the eueces. use 
of computer systems as patch solutions to, Sor, ‘human aimncutties. In the elma 
vein, verification proofs deal with, only Ape, Superficial | symtoms of system 
incomprehensible - ‘program “mistakes.” aren 

We have every right to demand the highest.» standard of yf rellabiity from sys 
tams that we deal with, and to require juetifoation for. the use ) of a peor = 


tem; Program verification plays. a neces sary and useful ple, in satiotying these 


2 ete at rig? peat ooo: 


demands. | But it is. wise fo temomber thet e Gage tie rrects 


Gt FR Ef BS 


3 is not a loense 


bd : Perey jo iahsy 


to establish a computer system as an autonomous decision eating bie et bed 
capable. of Initiating acts. (but, not capable | of.  2ccepting responsiblity for them). 
Unless we understand a ovation and: are _willing,, to aoe, Fesponsibiiity for it, we 
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6.2: Psychological Factors of Programming 

My discussion in the preceding sections of this chapter has focused on the | 
mechanical component of computer systems; Gerald Weinberg, among others, has 
emphasized nontechnical aspects of systems. The underlying message In his book, 
The Psychology of Computer Programming, is simple and obvious but nonetheless — 
underemphasized: computer programming is fundamentally a human activity, and a 
lack of attention to the psychological aspects of ‘this activity and to the social 
- environment in which It takes place can result in the creation of computer programs 
which are undesirable from many viewpoints. 

’ Early on, Weinberg acknowledges the impossibility of writing “perfect" com- 
puter programs and the need for computer programmers to recognize the Hmitations 
of their work: "Thus, there are degrees : of meeting specifications - of 
‘working’ - and evaluation of programs must take the type of imperfection Into 
account."*5 ‘the acceptable degree of “conformance to formal program 
specifications should be made explicit in the course of designing a system. If the 
users of a system are made aware of the ways in which the actual system does 
and does not conform to the proposed specifications of system behavior, then they 
may make more informed decisions about their use of, dependance on, and trust in 
computer output. 

One of the main concepts that Weinberg discusses is that of “egoless 
programming" - the training of software workers “to accept their humanity - their 
inability to function like a machine — and to value it and work with others so as to 
keep it under the kind of contro! needed if programming Is to be successful ....— 


the problem of the ego must be overcome by a restructuring of the social 


886 oraid Weinberg, The Psychology of Computer Programming (New York: 
Van Nostrand Reinhold Company, 1971), p. 19. 
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environment and, through this means, a restructuring of the value “yen of the 
programmers in that environment. »86 The goal here is to » coeur stereotypes that 
portray computer programmers as solitary workers whoes expertise is best exer- 
cised in isolation Cor, at least, in Isolation from other people, though | not Hecesserly 
from computers). Weinberg thinks. ‘that programmers, shoud not be encouraged to 


37 


identify. Memacives too _personally . with the programs ee write, a since this | can 


discourage cooperation between _ Programmers, particularly in uncovering 1 progres 
“bugs” at an early stage of coding. Furthermore, Weinberg | pereves that egoless 
programming can result In faster average debuoeing Gime, more accurate estimates 
of the progress of a programming project as ae work k proceeds, and th the Secareiioh 
of more reliable software. Weinberg continues to de-emphasize the role of the eal 
tary (and potentially indispensable) programmer by atiessing team Involvement in 
setting goals. In addition, he advocates oe workers, who fe wel baal a 
shifting environment and are willing to work together. 
| tt is. clear that Weinberg’s major point is that Jefective communication 
between the difforent ‘members of a programming project and i peieah system 
workers. and users Is necessary if we are to strive for better quality computer aye: 
toms. This Key leave le given relatively low statue in a typical software project 
For instance, documentation is frequently considered to be among halo jenst Set 
tant tasks associated with the generation of a _hew y Compute system, and cortainly 
peripheral to. the “real” work. of designing ane Programming the overt: instead, 


Weinberg believes that it should be elevated toa (professional status of its own. 


36 y/0., pp. 66-57. 


87 this does not mean that programmers should not accept personal respon- 
sibility for their work, but rather that their Identification with their’ programs shoutd 
Ce ee See ee govt a 
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As is the case with most activities involving human beings, "what is needed in a 
programming. project is slow, careful communication."2% The social behavior of 
software workers may need to be modified to facilitate more productive communice- 


tion between them. 


6.3: Modern Programming Practices 
' Evidence, taken from existing computer applications, about the quality of 
current software Is aimost unlformly discouraging. Work in-computer system reliabit- 
‘ity and program verification is, for the most part, still in. the research stage of 
development, .and interest in improving: programming environments remains primarily 
acadenic. 7 - ; Sy eee: 
In spite of methodological improvements such as_ structured 
design and coding, chief. programmer teams, orline. program 
‘development systems, high level languages and data base 
managers ..., the delivered quality of :Jarge scale software, 
whether new or modified, remains disgraceful, except where the 
projects or the people involved -are specially chosen. . Per- 
vasive cynicism about software Is the justifiable consequence 
of the many situations where poor results follow long de- 
lays .... Technical panaceas have falled consistengiy in the 
past and promise to do so for the foreseeabie future." 
Since software maintenance currently accounts for more. than fifty percent (and 
sometimes as much as eighty percent) of a typical. data processing budget,40 
there is obviously a strong incentive for the production of software that works. 
Why, then, is the quality of current computer.software so poor? 
The first thing to examine In attempting to answer this question is the goai 


of software that "works." Historically, this has been translated to. mean nothing 


88weinberg, p. 109. 


39)... McNeil, “Adopting a System Release Discipline," Datamation, Vol. 26, 
No. 1 (January, 1979), pp. 111-112. 


40Oipid., p. 112. 
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more than software that produces a tolerable Spproxipaton of the desired outputs: 
The predominant emphasis on minimizing the cost of computer system generation 
(for instance, incentive pay for early completion) has encouraged the satisfaction 
of the "workability" criterion through means of “Guestionabie reliability Over 
designing a first installation of a new system (building in - pert sigouphis some 
redundancy in the system - higher levels of Nene: rarenlty, watery, etc. than 
may nereeesrey be needed, to handie unanticipated aificuttios) | Is not Immedietoly 
cost erectve: Issues like dependability and. ‘comprehensibiity are on0 term corr 
cerns not directly related to the market value of a system; because of this, com- 
puter manufacturers have often Opted not to devote much oe thelr energies ta to the 
refinement of failsafe and fallure-proof techniques. Flexibility, modifabity, and 
maintainability have been treated as poicaamtd meenarmretiin of auanty assurance; 
primary components. are those that relate te & ayston's s: immediate performance. 

The emphasis. in a computer syéten project has eiwaye been on coding the 
system software. Productivity indices typtoalty messure.tines of. code written or 
number of compitable modules: produced ina given time period. ‘Structured program- 
ming, walk-throughs, and other modern programming: ‘prectows are: ‘eceuaaina evi . 
dence of the current interest in software engineering, but this is still a fledgling 
discipline. -Some programmers feel that the enforcement -of these’ techniques 
succeeds only In réfidering the activity of programming: more inflexible by allowing 
no room for finding and cultivating one’s ‘own programming style. ~All too often, it 
appears that the motivatitig philosophy tehind attempts: to-apgrade the quality of 
oti ‘calls for the Implementation of techniques: with -low. overhead and. highly 
visible results, without much consideration for their long term value. 

One difficulty Is the single. minded emphasis on impraving: she. quality ‘of com 


puter programs, not gions as a whole. The achiovement of better software must 
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involve changes .in the operation of ‘computer based organizations (software 
development and maintenance Is a management issue, not a purely technical one) 
and in individual habits (/.e., the development of a software engineering attitude 
and not merely the use of appropriate coding techniques). The continual change 
that is the way of life for most of the computer Industry can be handied in more 
‘reliable ways. Software workers can avoid incremental patchwork on systems in 
production between scheduled releases of the system, managers can resist the 
‘pressures to have little improvements pasted onto the current release without 
going through a complete testing cycle, and users can recognize that they cannot 
demand new system features without paying for them with time and money.*' The 
common themes in.most current discussions of "the software problem" are the need 
for improved education, planning, and communication, invoiving managers, data pro- 
cessing analysts, programmers, operators, and users. Ultimately, producing better 
computer systems is a human problem. "in the long run the evolution of quality 
software depends. upon people, not on systems analysis techniques, programming 


languages, oF operating systems environments."42 


6.4: Criticism 
The most intriguing questions about incomprehensibility relate not merely to 
computer programs, ‘but to computer systems and the: social environments In which 
they exist. 
One can, with the same technology, design totally different out- 
comes by designing different social support systems .... The 


technology is the same, yet the pattern of use is highly dissimi- 
lar. The crucial decisions are- sociological, not technological 


41 ipid., pp. 112, 114. 
42 nid. 
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-... the... social questions frequently Involve a difference If 
not a clash ‘of values .... it is not a matter of ‘right versus 
wrong but of right versus right. This is what makes porel, deck 
sions so difficult. And thesé are the hard quéi : on 


We have seen that there are personal and ‘social effects of the use of computer 
systems, as well as technical effects. Lewis Miinford suggésts ‘reckoning up “the 
human disadvantages ‘and costs, to say nothing ‘of the ‘dangers, of our ieanaeed 
acceptance of the system itself. n44 

Autonomous technology offers what it does “on one’ condition: that one must 
not ... ask for [anything] that the system does not offer... . Onte one opts for 
the system, no further choice remains.**5 i order to regain control over ‘computer 
technique, we must continually criticize “the tegitifiacy Gf the technological ques- 
tion";46 we must consciously decide whettié?r or not tis Is ‘what we want. The 


i Only f¥'Wewiden our sphere of moral 


_ development of technology’ may be modifie 
choices by considering otter aiternatives. This"may be dotie in many ways; for 
instarice, by utilizing the computer as only 4 ‘partial sélution’ to certain problems 
("... the computer too could be applied not’ sipiy Wwteéraver ‘the “opportunity 
arises, but only where It is deemed in the bast interests of sodiaty**% or by turn 
Ing the positive aspects of computer applications to purposes other than the per- 


petuation of the technological system. 


op hat sept 


. » 48paniel. Bell, “Hard. Questions and.Soft: Minds: .A Reply. to..Weizenbaum," 
_ Chapter 21 In The Future Study on the Impact of Computers and Information Pro- 
cessing, ed. by Michael L. Dertouzos (Cambridge, MA: The MIT Press, in prase). ..... 


“4wumtord., "Authoritarian and Democratic Technics," p. 57. 
4S ipia. SE GR Monee Pio tT 


48 oeeph. Welzenbaum,, "On the. Impact. got ta on Saclety," Sci- 
ence, 176 (May 12, 1972), p. 612. 


47a»bbe Mowshowitz, The Conquest of Will: Information. Processing in Human 
Affairs (USA: Addison-Wesley Publishing Company, 1976), p. 82. 


98. Chapter 6: What to Do About incomprehensible Systems 


The very leisure that the machine now gives ... can be 
profitably used, not for further commitment to still other kinds of 
machine, furnishing automatic recreation, but by doing significant 
forms of work, unprofitable or technically impossible under mass 
production: work , dependant upon special skill, knowledge, 
aesthetic sense. 

Even in a technolegical world, it is possible to criticize the system, but we forfeit 


our right to. criticize if we allow technology to dictate its own course. 


6.6: Responsibility 
The necessity for a strong sense of individual moral responsibility in today’s 
society cannot be overemphasized. One starting point for.computer professionals 
might be the set of "Guidelines for Professional Conduct in Information Processing" 
set forth by the Association for Computing Machinery; in the preamble, the ACM 
urges the following: 
The professional person, to uphold and advance the honor, dig- 
nity and effectiveness of the profession in the arts and sck 
ences of information. processing, and M keeping. with high -stan- 
dards of competence and ethical conduct: Will be honest, 
forthright .ead. impartial; will serve. with joyalty his employer, 
clients and the public; will strive to increase the competence 
~ end prestige-of the profession; wil use. his special. | on my 
and skill for the advancement of humen “eee 
emphasis) a sae Se 
nesponsmiaty: -ts here. defined not. merely in @ narrow professional sense, but rather 
in a broad sense that. takes into: eccount the relationshipa between Information pro- 
ceasing professionals and employers, cilents, other professionals, and the public. 
What requires additional! thought is:the- meaning of responsibility. in.terms.of posst 


ble repercussions on, for instance, programmers. Questions as to whether or not 


48 umtord, “Authoritarian and Democratic Technics," p. 68. 


49Reprinted in C. C. Gottlieb and’A. ‘Borodin, Soolal issues in Computing 
(New York: Academic Press, 1973), pp. 236-237. 
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computer professionals may be sued or put In fan (for example) for the production 
of “unacceptable” computer systems ought to be considered. 

The most important step In regaining contrat over technology ts the recon 
tion that we alone bear responsibility for our technological’ ‘wystems, and we are 
bound - in technological endeavors no lese than in other. endeevors.- by the social 
ethics governing human behavior in- society. The employment of computer systems: 
in tasks Involving people Invotves a social decision. The field of computer science 
enjoys substantial public support and has significant impacts on.soclety;'K has a - 
social responsibility. Moreéver;‘this social responsibility caiinct:be denied - 
exercised by defailt; through ‘tacit approval of new teetmological systems, even if 


explicit judgments ‘are not made. 


... the undeftying goal of scidfice and technology: is to _ 
improve ‘the quality of life. This implies that those closest to 

the technology have a special obligation to question the wees = ——ti<itits 
ek ce Le Wulcwas 0 cic tases itseaerahen an 


In questioning ‘our r technolody, we: mast! Se ws articular role 
we play In the system. “... only the ‘onder reat ose dover has a real 
aicuitheacemans®? Testmelogical inevitability need not be ddcupted; individual 
a, and constRutes a necessary ‘Setep* towards a return: t 
yo pessibte, giver béuPagie’ aint Ineight; for man to 
deny technology tiie prétogative to‘formuttite mai’s quisutisns. = it ts: podsibte to ask 
human questions ‘and to fiid humane anawets"92.. 4 


decision making fs poe: 


a eritored i 


50 Gottlieb and Borodin, pp. 241-242. 
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6.6: Humanism 

It should be clear by now that the improvement of computer programming 
techniques is a significant area of current computer science research (and, to a 
much lesser extent, of present computer science practice), but that the emphasis 
of this chapter has moved .to the Improvement of humanizing 
“techniques":- actually, attitudes - to apply to the creation and use of computer 
systems. | 

. In current practice, humanizing a computer system js often done cosmett 
cally; for instance, by having the system ‘Communicate with users in. a soothing, 
Englistriike dialect. While this is helpful, meaningful humanization of computer sys- 
tems must go beyond superficialities; humanizing a system must Invoive injecting 
the human element into a technological system. 

... we had better map out a more positive course: namely, the 
reconstitution of both our science and our technics in such a 
fashion as to Insert the rejected parts of the human personality 
at every stage in the process, This means gladly sacrificing 
mere quantity in order to restore qualitative choice, shifting the 
seat of.authority from the mechanical collective to the human 
personality and the autonomous group, favoring variety and eco 
logical. complexity inatead of stressing .undue uniformity and 
standardization, above all, reducing the ‘Insensate drive to 
extend the system itself, Instead of containing Ht, within definite 
human limits .... We must ask, not what Is good fpr science 
or technology, . . . » but what is good-for.man...."" 

At the very least, humanization requires a. recognition of human values. Com- 
puters. may be used to explore alternative courses of action, but values must be 
included in decision making about the use of computerized "answers." "What is evi 
dently wanted Is a set of balance sheets in which the relative moe ot each solu- 


tion to a technological problem are analyzed both on n technological grounds such as 


53 yumtord, “Authoritarian and Democratic Technics," p. 58. 
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safety, ease of operation, complexity, and esthetics, and on ethical grounds such 
as moral considerations, effects on the quality of human life, liberty and dignity, and 
other human values."94 We must come to believe thet the dictates of human 
beings supersede those of technological systems; there is relatively little compel 
ling evidence that modern ‘society does belleve idea in addition, a strong sense of 
control over the use of technology should be ‘nurtured; bs seated this should 
involve “cut[ ting] the whole system back to | a pom at which It will permit human 
alternatives, human intanientiona; and human destinations’ for entirely different Per 
poses from ee of the system itself. n55 : . 

In an organizational context, care et be taken to avoid undesirable efacts 
of the increased rigidity, formality, ant alienation ‘that frequently denied com — 
puterization. One should not igure the nonrational aepects of social conduct by 
administering a social organization sash Re to Purely ecece criteria. “The 
modern bureaucracy is largely responsible td hia pursutt of efficiency - indifferent 
to human needs and unsupportive of the Promotion oF interaction and Supeuceso ; 
across hierarchical leveis. ‘Mowshowitz points ha the’ ctoutty “of ohatenging the 
status quo: "... one critical feature is a mochartort toe establishing a continuing . 
dialogue between managers and workers: or ordinary chttens : This ind not likely to 
emerge ‘spontaneously, since it presupposes a fundamental nine In ‘vahrwe from pro- 
ase as oie: especialy important 
in what Mowshowltz and others have’ sald: about imprdving coriditions ‘in’ a téethnolog- 


ductivity and eMciency to hutian welt 


Ical society. “First, communication Is seen as « key faétor in chaiging theextating 
situation; One aspect Of this Is that ‘more ‘of ‘the people who interact with: computer 


S4vowshowitz, pp. 271-272. 
‘66 yumtord, “Authoritarian and Democratic Téchnics," p. 69: 
56 uowshowitz, p. 201. 
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systems shouid have access to Information about the system, as well as the power 


to use this information. Second, perhaps the ultimate deterrent to comprehending 


computer systems is the human value system that places auch a premium on tech . 


nology. 

Finally, and most importantly, technological system incomprehenslibility Is a 
human problem, and the most significant and difficult questions that it raises should 
be answerable to humanistic concerns. Any meaningful comprehension of our 

’ Interactions with computer systems must be preceded by a better understanding of 
our own role in a technological society, and this in turn requires an understanding 
of interpersonal difficulties, human priorities, and ethical values. 

What should this teach us, particularly with respect to the 
question of at least preserving If not enhancing human choice in 
human affairs? 
Certainly that the construction of reliable computer 
software awaits, not so much results of research In computer 
science, but rather a deeper theoretical understanding of the 
human condition. 
Before computer systems can be made truly comprehensible, human systems must 
‘be better understood. Before we can control our technological systems, we must 


learn to value people more than technics. 


57 joseph Weizenbaum, “Human Choice in the interstices of the 
. Megamachine," p. 14. Lecture presented at the IFIPS Conference on "Human 
Cholee and Computers," Vienna, Austria, in June, 1979. 
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